Evaluating socio-economic programmes : volume 5, Transversal evaluation of impacts on the environment, employment and other intervention priorities.
EC Structural Funds iluating socio­economic programmes Transversal evaluation of impacts on the ¡bther* ■¡ties MEANS Collection Volume 5 opean Commission EC Structural Funds Evaluating socio-economic programmes Transversal evaluation of impacts on the environment, employment and other intervention priorities Volume 5 European Commission MEANS is a programme of the European Commission. It is aimed at improving Methods of Evaluating Structural Policies. The MEANS Programme and Collection was supervised by Phillipe Goybet and Miguel Angel Benito Alonso, Unit for the Coordination of Evaluation in the Directorate for Coordination and Evaluation of Operations of DG XVI Coordination-evaluation@fmb. dg16. cec. be The realisation of the Programme was entrusted to the Centre for European Evaluation Expertise (C3E) under the direction of Eric Monnier and Jacques Toulemonde. Responsibility for the opinions expressed in this book lies solely with the authors of the text: C3E - 13 bis Place Jules Ferry - F - 69006 Lyon - France - c3e@c3e. fr A great deal of additional information on the European Union is available on the Internet. It can be accessed through the Europa server (http://europa. eu. int) and on the Inforegio Website (http://inforegio. cec. eu. int). Cataloguing data can be found at the end of this publication. Luxembourg: Office for Official Publications of the European Communities, 1999 ISBN 92-828-6626-2 © European Communities, 1999 Reproduction is authorized, except for commercial purposes, provided the source is acknowledged. PRINTED ON WHITE CHLORINE-FREE PAPER Printed in Italy Foreword Developing good evaluation practices has become a priority area for the is directly relevant Commission and to the effective use of Community resources. Indeed, the sound financial management principles which the Commission seeks to promote via its SEM2000 (Sound and Efficient Management) Programme rely, specifically, on the regular evaluation of the Community activities. The Directorates General responsible for structural policy have considerable expe­ rience in this area. Since 1988 they have made evaluation an integral part of each stage in the process of Community Structural intervention. Thus, the programme development stage includes a prospective evaluation; the implementation stage a strengthening of the monitoring systemsand, at the end of the intervention, evaluation establishes what has been achieved in terms of programme impacts. However, the capacity of evaluation to contribute to the effectiveness and transparency of structural interventions rests ultimately on the integrity of the methods adopted to verify the credibility of results. This is why, four years ago, on the initiative for the Directorate General for Regional Policy and Cohesion (DG XVI), the MEANS programme was initiated as a response to the demand for developing evaluation methodo­ logy. This also include practices combining both general scientific requirements and the specific requirements of Community interventions whilst, at the same time, remaining accessible to a large audience. One of the more tangible results of the MEANS programme has been the publication of this six volume set by a group of independent experts, focu­ sing on the principal elements of socio-economic programme evaluation. A short summary of this work can be found at the beginning of the present volume. This collection covers the various methodological approaches and innovations that have taken place in the context of structural funds eva­ luation. It will enable those responsible for evaluation to develop their approach in a flexible and independent manner and for the various programme partners to exercise greater control over the evaluation pro­ cess. Attending the conferences of evaluation specialists and "clients" convened in Brussels (1995), Berlin (1996) and Seville (1998), I gained some impression of the interest that new evaluation techniques can gene­ rate. I would stress that the development of evaluation needs to progress in tandem with efforts to incorporate evaluation findings in the decision making process. This implies evaluation as a means of establishing whe­ ther policies are equitable, effective and relevant to the identified needs. In satisfying these requirements, evaluation should be seen as a valuable instrument affording both Member State and Community authorities the necessary guarantees that Community funds are being deployed in the best interests of the EU citizen. This is a cost worth paying when the goals of Economic and Social Cohesion and the European Union enlargement are at stake. Finally, I wish to extend my thanks to the members of the Committee of Independent Experts established at the launch of the programme. Their expertise and standing have contributed greatly to the quality and relevan­ ce of the MEANS programme. I therefore thank: E. Chelimsky (USA), ex- President of the "American Evaluation Association"; K. Kennedy (IRL), Director of the "Economic and Social Research Institute" (ESRI); J. R. Cuadrado Roura (SP), President of the "European Regional Science Association"; C. Seibel (F), Director of the "Statistiques démographiques et sociales de l'Institut National de la Statistique et des Etudes Economiques" (INSEE); L. Senn (I), Director of the Department of Economic Regional of the University Bocconi; E. Stern (U. K. ), Director of the unit of evaluation of the Tavistock Institute; A. Sorber (t) (NL), Head of the "Policy Analysis Department of the Ministry of Finance"; L. I. Strom (SV), Statens Institut för Regionalforsking (SIR); L. Tsoukalis (GR. ), professor at the University of Athens; H. Wollmann (D), professor at the University of Humboldt - Berlin and founder member of the "European Evaluation Society". E. Landaburu The MEANS Collection: "Evaluating socio-economic programmes" The evaluation of European Union structural interventions has entered into a phase of maturity, attested by the extension of work carried out by both the Member States and the Commission. Regu­ lations based on decisions by the Council have largely contributed to the development of evaluation practice and have stimulated methodological research. Over the past ten years, the Commission has striven to promote and improve methods for evaluating Structural Funds. The main metho­ dological conclusions have been debated and disseminated as part of the MEANS programme initiated by the Commission in 1991. Achievements to date provide enough material for a complete and accessible publication intended for a broad public: a six volume Collection covering all aspects of the evaluation of socio-economic programmes. These Volumes are written with a perspective that is broader than the current administrative framework, for this is likely to evolve more rapidly than professional standards. In general, the Collection Is aimed at a readership of non-specialists seeking essential information on the evaluation of socio-economic development programmes. These programmes have multiple objec­ tives, a partnership dimension and a multi-sectoral content - all cha­ racteristics which call for suitable evaluation methods and solutions not yet provided by the international technical literature. The complete Collection consists of 6 volumes designed to be comple­ mentary and to provide essential information and recommendations for professionals wanting to independently develop their own evalua­ tion practices. Each volume can, however, be read and understood without having to refer to the other volumes in the Collection. The first Volume in the MEANS Collection entitled "Evaluation design and management" serves as an introduction both to the Col­ lection and to the evaluation of socio-economic programmes. It enables readers to understand the implications of evaluation and to know what they can and cannot expect from an evaluation at a given time in a particular context. It covers all the elements needed to steer an evaluation project from A to Z: sharing of responsibilities, defining of the aims and method of an evaluation, establishing a schedule and budget, assessing the quality of the work performed and, finally, managing the use of conclusions. The second Volume focuses on the "Selection and use of indicators for monitoring and evaluation". It deals with the subject of indicators used for monitoring and evaluation. It shows how indicators can be used to support policy decisions, improve management and measure the overall performance of programmes. The volume proposes a methodological framework and a practical guide for the use of indicators in these different situations. The third Volume presents a selection of "Principal evaluation tech niques and tools" applicable to the evaluation of socio-economic programmes. These tools are taken from relevant scientific disci­ plines: economics, sociology, management, geography, etc. Each one of the 23 tools Is described in the form of an information sheet featuring a general description, the principles of its implementation, a list of its strengths and weaknesses for the evaluation of Structural Funds, an example of its application and a list of references. A number of methods adapted to the evaluation of socio-economic programmes are the object of a more detailed description in the fourth Volume entitled "Technical solutions for evaluation within a partnership framework". Several pilot evaluations were implemen­ ted within the MEANS programme to design and test the following four innovations, the application of which is described in detail in this Volume: impact mapping, cross-impact matrix, score cards and multi-criteria analysis. The fifth Volume deals with "Transversal evaluation of impacts on the environment, employment and other intervention priorities" of the European Union and Member States. Among the questions addressed are impacts in terms of equal opportunities between men and women and the competitiveness of SMEs. The Volume provides the elements required for an analysis of those impacts which are not necessarily part of the explicit objectives of all the actions evaluated. Finally, the last Volume is a "Glossary of concepts and technical terms". The compilation of the Collection has necessitated the rigorous standardisation and definition of the terms used. Each of the first five volumes has an index of terms which, along with hundreds of definitions, are brought together in the glossary. Contents Introduction I Technical considerations relating to the assessment of thematic impacts II Thematic analysis of impacts on political priorities Evaluating the environmental impact of structural interventions Evaluating the impact of structural interventions on the competitiveness of Small and Medium-Sized Enterprises Evaluating the impact of structural interventions on employment Evaluating the impact of structural interventions on equal opportunities III Thematic evaluations of domains of interventions Evaluating impacts arising from interventions in the domain of Research and Technological Development Evaluating impacts arising from interventions in the domains of transport, telecommunications and tourism IV Framework for performing thematic evaluations A step-by-step approach for thematic analysis Cross programme thematic evaluation Conclusion Appendix 1 - Evaluation tools and corresponding themes Appendix 2 - Extracts from a Terms of Reference for a programme evaluation incorporating thematic elements Appendix 3 - An example of terms of Reference for a transnational thematic evaluation Appendix 4 - List of boxes Index 7 13 27 29 59 77 107 129 131 151 167 169 179 189 195 197 199 209 213 Introduction Economic and social cohesion is the global objective of programmes financed by the Structural Funds. The objective has three elements: catching up of lesser-developed regions, social and economic reconversion of urban and rural areas and reintegration of social groups in difficulty. Interventions implemented by way of these main objectives must try to satisfy four political priorities common to the European Union and the Member States (see Box 1). These priorities are most often recalled in the preamble of programmes. However, axes entirely devoted to them within a programme are not neces sarily found. A sub-programme, an environmental priority, or even specific measures concerning the competitiveness of SMEs are sometimes to be found. But countries or regions have rarely programmed specific measures focusing on equal opportunities. Transversal evaluation of impacts on intervention priorities Box 1 - Priorities, domains and themes The political priorities defined for the 1994-99 programming period of the Structural Funds comprise: • the fight against unemployment and the creation of new jobs; • increasing the competitiveness of productive structures, more particu­ larly SMEs; • protecting the environment; • and finally, reducing inequality between men and women. The notion of "domain" refers to specific domains in which interventions take place and, therefore, outputs are programmed. Thus, for example, impacts in terms of "competitiveness" are the consequence of interven­ tions in the domains of research and development, transport, telecom­ munications, etc. The notion of "theme" is used in this book to cover both political priorities and domains of intervention. A thematic evaluation might therefore examine if a priority has been respected in all of the domains of interven­ tion. On the other hand, another thematic evaluation might be interested in a given domain and examine all of the possible impacts (notably for the four priorities mentioned above). The four priorities, which differ in importance according to the context, are transversal in relation to the programmes financed by the Structural Funds. All managers and operators are supposed to apply them in the implementa­ tion of measures, in project selection, in the definition of target publics, etc. Most often, the evaluation of a programme favours an analysis of impacts, which are explicitly described by the principal objectives of priorities and measures. It tends to neglect transversal priorities, which concern, in a way that is less precise, the entirety of programme measures. Progress is therefore necessary if reports are to contain reliable thematic conclusions at a reasonable cost. These conclusions must be presented in a standardised and sufficiently synthesised form in order that the Commission Services can themselves use them to produce a European-level synthesis. Moreover, in accordance with the principle of subsidiarity, the Commission has to periodically evaluate the different modes of intervention applied in the Member States in order to identify best practices and organise the transfer Introduction of know­how in Europe. This is done through evaluations performed at the European level, with a domain of intervention (transport infrastructure, support for research activities, professional training, etc. ) being subject to an in­depth analysis. These domains of intervention are generally found, to greater or lesser degree depending on the case, in the majority of regional programmes. ■ A Volume for Community and programme level commissioners of evaluations This Volume is destined for those who, at the programme or European Com­ mission level, want to undertake a transversal examination of impacts cor­ responding with a priority or domain of intervention. These "thematic" evaluations are designed differently according to whether they are performed at the programme level itself or the European level. At the programme level, thematic evaluation covers both the direct impacts of measures devoted to a given priority (a measure in favour of the environ­ ment, for example) and the indirect consequences of other measures (envi­ ronmental consequences of road building, for example). In relation to a given theme, this first type of evaluation is a synthesis of the entirety of possible impacts, both direct and indirect, and generally verifies the overall coherence (and the mutual reinforcement) of the different measures contained in a programme. The conclusions of this type of evalua­ tion are often placed in a series of chapters dedicated to each of the themes at the end of an evaluation report. The second type of thematic evaluation, for which the Commission Services are responsible, is to produce conclusions relevant for a number of regions or countries of the European Union. These evaluations commence with a "trans­programme "analysis of the impacts of similar interventions in order to identify the most efficient, and to distinguish, in the case of a successful intervention, between what is proper to the economic, administrative or ins­ titutional context and lessons that can be applied to other territorial, regional or national units. This type of evaluation highlights the relative efficiency of interventions and plays a central role in the identification of best practices. This Volume contains advice and solutions targeted at public officials who want to include these themes in their terms of reference and evaluation Transversal evaluation of impacts on intervention priorities teams wanting to design a work programme (see examples of terms of reference in the appendix). In order to take on the challenge of obtaining reliable thematic conclusions at a reasonable cost, in the following pages analytical grids and methodologi­ cal solutions can be found for: • identifying potential impacts of a programme in relation to a theme ; • transversally taking into account impacts within the same programme or within group regional or national programmes; • identifying and defining suitable indicators for monitoring and evaluating these impacts; • collecting and processing appropriate information; • and finally, presenting a synthesis of the conclusions of an evaluation. ■ T he structure of t he Volume This Volume does not claim to deal with specialist concepts and methods relating to the themes tackled. It could easily be imagined, for example, that experts in employment policy would consider that the subject has not been sufficiently developed, or that environmental specialists would judge that the analysis of ecological impacts is highly superficial. Indeed, the raison d'être of this volume arises from the unrealistic demands of experts of different themes concerned by structural programmes. The challenge for programme evaluation is to analyse a programme's merits in respect to its own objectives and in relation to the four political priorities common to the European Union and the Member States. For reasons of time and budget, an approach which simplifies each theme has be taken. In order to do this, concepts and solutions specific to each area of expertise have to be selected, and a trade­off has to be made between reliability and feasibility. The text is organised in four parts. General technical considerations appli­ cable to all thematic evaluations are brought together in the first part of the Volume, so as to highlight the points common to all themes. It should be noted that the development of certain points is limited and these points are more fully developed in the first and third Volumes of the Collection. Volume 1 of the MEANS Collection describes the entire evaluation procedure and Volume 3 presents a set of tools for the collection and processing of informa­ tion. Introduction The second part of the book is given over to the principle political priorities: environment, competitiveness of SMEs, employment and equal opportuni­ ties. Methodological requirements vary from one priority to another. Thus, the taking of environmental impacts into account in an evaluation is based on methods adapted to the estimation of impacts that are diffuse and hete­ rogeneous. On the other hand, employment impacts are by nature more homogeneous since they relate to people and enterprises. For example, a survey of direct addressees is suitable for studying employment impacts, but not environmental impacts. The creation and maintenance of employment has, since the start of the 1980s, been one of the preoccupations of the European Union. This is concretely demonstrated by the numerous measures devoted to employ­ ment within programmes financed by the Structural Funds. Moreover, a large amount of conceptual and technical work has been undertaken on this theme by research centres, allowing a substantial quantity of know-how to be accumulated. Within the framework of the MEANS Programme, numerous workshops and reports have been given over to the theme of employment. This Volume emphasises the selection of realistic and uncom­ plicated solutions within the framework of thematic evaluations and cannot claim to be situated at the same level of sophistication as an actual evalua­ tion of employment policy A theme such as equality of opportunity between men and women has given rise to important conceptual advances, for example, the notion of gender mainstreaming (Box 2). Box 2 - Gender mainstreaming: a definition Gender mainstreaming is the systematic inclusion of the priorities and needs specific to women and to men in all Community policies and actions, with the unique aim of achieving equality. This is to be achieved by actively and visibly taking account, at the planning stage, of the effect of interventions on the specific situations of women and men during their implementation, follow-up and evaluation. Gender mainstreaming requires the development of measures aimed at adapting the way society is organised to ensure the more equal distribution of roles between women and men. Transversal evaluation of impacts on intervention priorities However, evaluations that tackle this theme are still few, reflecting the fact that this preoccupation is relatively new for the majority of countries. This type of thematic evaluation is more often ex ante than ex post. The examples of evaluations presented are therefore aimed at prospective rather than retrospective analysis. In order to prepare this chapter it has therefore been necessary to go beyond existing practices. The third part of the text deals with thematic evaluations of domains of intervention. Four domains have been selected not for any strategic reason, but more prosaically because they have recently been the object of European level thematic evaluations within the framework of the Structural Funds. The domains of research and development, transport, telecommu­ nications and tourism are examined. The first of these domains is tackled in some detail, while the others are more briefly examined. The Volume 2 of the Collection provides a more complete view of all the domains of inter­ vention from the point of view of indicators for monitoring and evaluation. Part IV of the Volume focuses on methodological issues which cut across thematic boundaries and which are specific to evaluations conducted at the programme or EU level. Firstly, a standardised procedure is provided which is designed to structure the implementation of programme level eva­ luations in such a way as to ensure that conclusions can be aggregated and synthesised at a supra-national level. Secondly, the basic principles of transnational thematic evaluation, organisational issues associated with this type of evaluation and its use in highlighting best practice are examined. Finally, a possible future direction for transnational evaluation practice is mapped out. I Technical considerations relating to the assessment of thematic impacts The evaluation of the impacts of Structural Fund interventions is structured around a number of basic, but very important, tasks which need to be under taken, whether the evaluation is focused on a theme, a political priority or an entire programme. Some of these key tasks include: the clarifying of explicit and implicit objectives, selecting indicators, gathering data, and estimating net impacts. The objective of this chapter is to present how these four tasks can be approached. This presentation is not theme specific and is illustrated by examples drawn from the individual themes presented further in this book. The issues tackled are, of course, applicable to themes that are not presented here, but which may be taken into consideration in or be the object of future evaluations. Technical considerations ■ Clarifying the explicit and implicit objectives of t h e me focused interventions An early step in an evaluation is the clarification of a programme's objectives. During the ex ante evaluation of a programme it is important to clarify not only its central or global objectives, but also those objectives related to specific priorities, such as those mentioned in this book. In intermediate eva­ luations, additional objectives that might have emerged in the course of the programme should be clarified, along with the contribution of projects to attaining objectives. Even if there is a single overall priority objective (e. g. the promotion of envi­ ronmentally sustainable development), attaining that objective is likely to depend on attaining specific and intermediate objectives that may be expressed differently. It is therefore necessary to draw up a clear hierarchy of the objectives and the relationships between them. In order to draw up the hierarchy of objectives an "objectives tree" can be constructed. This tool allows the degree of internal coherence between the measures of a programme in relation to overall priority objectives to be examined (both vertically ­ between different levels of objectives ­ and hori­ zontally ­ between the objectives of different components of the programme which are at the same level). It also facilitates the examination of the external coherence of a programme with other public policies. For example, Structu­ ral Fund intervention in favour of SMEs might target the development of innovatory capacity, increased networking and technology transfer, while many Member states are seeking to create a general business environment which is conducive to the development of small firms. An examination of external coherence would focus on whether the objectives of these interven­ tions were compatible or not. While the hierarchy of objectives is usually constructed on the basis of official documents, in the case of thematic evaluations it can also be useful to incorporate implicit objectives into the analysis. An objectives tree constructed in this way can therefore be considered as a thematic objectives tree. Although the incorporation of implicit objectives into an analysis allows a better understanding of the logic of a programme, it is important to be careful when using implicit objectives as evaluation criteria. When the work of programme managers and operators has to be judged, this should only be done in relation to explicit objectives of an intervention. Transversal evaluation of impacts on intervention priorities The principle of a thematic objectives tree is illustrated in Box 3. Actions financed by the Structural Funds are designed to attain the explicit global objective of social and economic cohesion. A hierarchy of the explicit objectives linking actions to the global objective can be constructed in the form of an objectives tree. At the same time the promotion of equality between men and women is a priority concern for Structural Fund programmes, but objectives related to this political priority are largely implicit. While global objectives are stated in official texts, working documents and studies, this is usually not the case for specific and intermediate objectives. These objectives are often constructed implicitly by the various actors of the programme and serve to regulate and direct their operational decisions. The hierarchy of implicit objectives is presented in a thematic objectives tree which allows the relations between actions and the global objectives of a political priority to be structured. It should be remembered that the objectives trees presented in the following chapters are only illustrative, objectives trees should be constructed by the evaluation team in consultation with the steering committee of the evalua­ tion on the basis of the specific programme to be evaluated. Moreover, an Box 3 ­ Using objectives trees for clarifying implicit and explicit objectives related to a given theme Global objectives Intermediate and specitic objectives Actions ■* Objectives tree (of a programme) Social and economic cohesion Balanced development for men and women Thematic objectives tree Explicit objectives Implicit objectives Technical considerations objectives tree is neither exhaustive nor definitive and it will have to be modified over time to take into account the evolution of political priorities and the socio­economic context. For more information on the clarification of objectives using an objectives tree, see Volumes 1 and 4 of the MEANS Collection. ■ Selecting indicators for thematic evaluations Volume 2 of the MEANS Collection is devoted to the subject of indicators for monitoring and evaluating Structural Fund interventions. Only essential elements are developed hereafter. In order to assess the impacts related to a given theme, indicators need to be defined which provide an estimation of the phenomenon to be measured. The selection of several complementary indicators may be necessary to estimate an impact that is considered as important. For example, in order to estimate air quality, a number of indicators relating to different types of atmospheric emissions would have to be employed. Given the multiplicity of objectives, the number of components of a programme (sub­programmes, measures, and actions) and the need to take into account transversal political priorities, it is difficult to imagine an opera­ tional system of indicators which would include indicators for all compo­ nents at all levels. It is therefore necessary to select indicators that meet the most important needs of managers and officials and can be collected at a reasonable cost. In the case of thematic evaluations, since the role of indica­ tors is to allow an intervention to be judged in relation to the political priori­ ties to be taken into account, they should be related to the most important impacts of the theme concerned. The indicators retained should be reliable, giving the same indication irrespective of the person quantifying them and providing up­to­date information on the effects being measured. The construction of a system of indicators for performing thematic evalua­ tions requires, first of all, that the relevant evaluatory questions are establi­ shed. These questions will usually encompass the following issues: "To what extent have changes taken place?", "To what extent are these changes attri­ butable to the programme?" "Are the results and impacts that have occurred satisfactory?" Secondly, those evaluation criteria by which an intervention will be judged are agreed. Thirdly, that context indicators, which provide information on the reference situation in which an intervention occurs, and Transversal evaluation of impacts on intervention priorities programme indicators, which focus on the direct and indirect effects of an intervention its addressees, are defined in the light of the evaluatory questions and criteria selected. The in-depth analysis of impacts also requires that exogenous factors are taken into account, i. e. those factors over which the programme has no control and which have an influence on impacts. It is therefore useful to identify the main effects of confounding factors by means of indicators. A system of indicators is designed to allow the tracking of an intervention from the resources employed during its implementation to its impacts as far as possible along the chain of causality. In order to do this indicators furnish information on the human and financial resources employed in implementing an intervention (resource indicators), the material and immaterial outputs produced (output indicators), the immediate impacts on the addressees resulting from an intervention (result indicators), and the subsequent impacts on direct and indirect addressees (impact indicators). When referring to different types of indicators and the concepts associated with them, it is highly desirable that a standardised terminology be used. It is therefore strongly recommended that the definitions figuring in Volume 2 of the MEANS Collection are systematically employed. Example: Indicators for measuring the effects of interventions for improving water quality and resources In order to improve and safeguard the quality of local water supplies, a programme of investments is planned with the objective of increasing the amount of waste water treated and reducing the amount of treated water lost through leakage. At the output level indicators would be chosen that measure the physical achievements of the inter­ ventions, for example: kilometres of mains supply pipes modernised, the capacity of new waste water treatment plants constructed and the capacity of waste water plants to undertake primary / secondary treatment. An indicator at the result level is the number of households or businesses connected to the plant. At the impact level indi­ cators would include the volume of water consumed via the modernised system, number of days of when supply is restricted and volume of water lost through leaks. Impact indicators also reflect improvements in the quality and quantity of underground water resources and river quality, for example: withdrawal rate from sources, withdra­ wal rate as a % of annual availability, % of waste water samples complying with EC directives, % of length of rivers monitored subject to slight or moderate pollution. It is necessary to take the timescale of effects into account, otherwise judging them will be distorted by incomplete information. This will result in Technical considerations the underestimation of expected effects that have yet to occur and / or the overestimation of temporary effects that have not come to an end. Example: The timing of employment effects arising from the promotion of research collaboration A project to improve collaboration between SMEs and public research bodies would produce different employment effects at different time scales. In the short­term, employment effects would be mainly temporary with employment being maintained or created during implementation (e. g. advisors, consultants, etc. ). In the more medium­ term the recruitment of technicians, researchers and scientists to take part In the project would lead to permanent employment of individuals associated with the per­ forming of research. In a longer­term perspective employment Impacts will occur within the SMEs which are involved in research projects as the resulting innovations are marketed and levels of production increase. Four basic features of the indicators should not be forgotten in the interpre­ tation of data. First, the information provided might require additional pro­ cessing, since it is not always usable or communicable in its initial state. Secondly, the changes measured may not be entirely caused by the inter­ vention being evaluated and in any event, the analysis of an indicator is not enough to establish relations of cause and effect. Thirdly, when comparing information provided by an indicator, it is important to ensure that the comparison is relevant (i. e. that the context of the reference situation is the same). Finally, indicators should also provide infor­ mation on confounding factors which influence the evolution of the reference situation (e. g. when evaluating the impact of a job creation programme on unemployment, changes in the size of the working population should be taken into account). ■ Gathering data about a t h e me When the evaluation of a theme is performed within the framework of a programme, it is recommended that this is based on documentary research and existing data (e. g. monitoring data, statistical surveys and existing eva­ luation reports) supplemented by minimal case studies, interviews and focus groups (see the "step by step", approach presented in Part IV). Transversal evaluation of impacts on intervention priorities Box 4 - In-depth analysis of impacts: appropriate tools and constructing an evaluation method When an in-depth analysis of impacts is required, it is necessary to construct an evaluation method composed of a number of steps (a "technical chain") which includes a combination of interdependent tools for primary data collection and analysis. The "step-by-step" approach for thematic evaluations, proposed in Part IV of this Volume, while remaining faithful to the "technical chain" concept favours the use of a limited range of tools that facilitate a "quick and dirty" approach to primary data collec­ tion. Some tools, more often associated with in-depth analysis, are, however, better suited to certain types of thematic evaluations than others. It is with the objective of highlighting the suitability of certain eva­ luation tools to different themes that tools for in-depth analysis are presented in each of the chapters devoted to specific themes. Volume 3 of the MEANS Collection provides a full discussion about the choice and use of evaluation tools within the framework of the evaluation of Structural Fund interventions. The monitoring of the implementation of interventions makes most use of resource, output and result indicators. These are largely quantified by programme managers and their services which issue payments and create or collect information on: number of outputs and / or level of physical output; expenditure; descriptions of the projects; reports on activities of manage­ ment services. Several services are usually involved in this activity, so it is generally necessary to centralise these data in advance, concentrating on the most useful data for managing and evaluating interventions. If this basic information is to be suitable and complete, the collection of monitoring infor­ mation should be planned from the outset, bearing evaluation needs in mind. It is not always possible, however, to supply data for indicators through existing information systems. This is, in fact, the case for the majority of indicators focusing on the short or medium term impacts of structural inter­ ventions. The use of statistical time series data, while relatively cheap and quick, has a number of major drawbacks which limit their use in evaluation: the time-lag inherent in the production of data for standard statistical surveys; the absence of a direct link between an evaluation criterion and available statistics, the dif- Technical considerations ference between the geographical area covered by an intervention and the ter­ ritories or groups of territories used to compile statistics. Regional statistical observatories have also been created but, even when statistics are presented at the level of territorial sub-divisions of the Member States, these sub­ divisions do not always correspond with zones eligible for Structural Funds. In these conditions, the trends highlighted by the statistics may not allow direct conclusions to be drawn on the effects of a particular programme. Statistical information systems are, however, being developed to remedy this shortcoming. Eurostat is co-ordinating the collection of data used to provide information for indicators of the activity of 35 industrial sectors in all the areas eligible for support to promote development, structural adjustment and industrial reconversion. These data are available for the period 1985- 1993 and will be available for 1994-1999. The different information sources have various strengths and weaknesses that are summarised in Box 5. Box 5 - Typical strengths and weaknesses of sources of existing information Strengths Weaknesses Monitoring systems Produced at frequent intervals Relatively up to date Generally readily available Significance Is usually apparent Relatively low cost Permanent statistical surveys Low cost Reliability Richness Comparability Ease with which information Is obtained Other research and evaluations Low cost Data already analysed Sometimes unreliable statements Rarely available for all measures / actions Data on physical outputs and results of various components not easily aggregated Not easily comparable Often not directly linked to evaluation criteria Geographical level not always suitable Information not up to date Information not up to date ften too restricted to provide a reliable answer or one that can be generalised Given their strengths and weaknesses, the sources of information are com­ plementary rather than alternative. For example, permanent statistical surveys provide data for comparison (e. g. evolution of employment at regional and national level), and indications of the evolution of the context Transversal evaluation of impacts on intervention priorities (e. g. economic situation, evolution of workforce, growth and decline of certain sectors) which allow for a more in­depth analysis of impact indica­ tors, and in some cases, relevant impact indicators at a reasonable cost. On the other hand, evaluations and existing research provide elements for com­ parison (effects of similar interventions), elements for estimating deadweight or displacement coefficients allowing the calculation of net impacts, and hypotheses to be postulated on ways in which effects are produced. ■ Estimating net impacts The need to estimate the net impact of an intervention applies to all evalua­ tion themes, however, Indicators are only capable of providing information on the gross impact of an intervention (i. e. the observed change in the reference situation or the situation of direct and indirect addressees). The gross impact of an intervention is not a proxy for its net impact and, as a consequence, the latter has to be estimated for the evaluation to provide accurate information. The estimation of gross impacts is therefore only an initial step in the evalua­ tion of impacts that can be genuinely attributed to an intervention. While the factors that have to be taken into account when calculating net impacts depend to some extent on the theme of the evaluation and the indicators employed, the following convention represents a minimal definition: Net impacts = Gross impacts ­ Deadweight ­ Displacement (Substitution)* ('Unintended or unwanted) Deadweight is the term given to the change that would have been experien­ ced even in the absence of an intervention. Assessing displacement is relevant for interventions targeted at geographical zones. On the other hand, assessing substitution is relevant in the case of social group­targeted inter­ ventions. Only when displacement and substitution are not objectives of an intervention are they subtracted from gross impacts. In order to assess the significance of deadweight, it is necessary to construct the counterfactual situation or policy­off scenario based on detailed projec­ tions of the likely evolution of key variables if no intervention had occurred. The reconstruction of the counterfactual situation is of critical importance for the accurate assessment of the impacts of structural interventions. It is, however, a difficult task that becomes even more complex at programme Technical considerations level when a variety of inter-related interventions has to be considered. In practice, there are very few occasions when the counterfactual situation can be reconstructed with certainty; an open, reasoned and informed effort to make a best estimate is frequently the best that can be done. An illustration is given in Box 6 of the principle of deadweight. In this example the change in the number of jobs during the period of an interven­ tion (from baseline to the moment of the evaluation) is j0n less jo, in other words the gross employment impact. During this period, however, it is estimated that in the absence of the intervention, the policy-off situation, employment would have increased to j0ff- The impact of the intervention net of displacement is in the difference between j0n and j0ff. Box 6 - Estimating impacts net of deadweight 1 on J off Pollcy-on </ A , A ;/ ^^-^^ Policy-off Impact net of deadweight y k Deadweight Change of situation (gross impact) γ τ y Baseline Evaluation Τ When displacement occurs, impacts on addressees are accompanied by effects on non-addressees within the programme area. An example of this is when a reduction in traffic-related atmospheric emissions due to the construction of a new road leads to the unintended displacement of road traffic and exhaust emissions to another part of the programme area. In this instance the effect is unwanted and displacement should be subtracted Transversal evaluation of impacts on intervention priorities from gross impacts. If, on the other hand, displacement is sought as an impact of an intervention (e. g. the movement of small-scale industrial firms causing noise and atmospheric pollution from a heavily populated city area to a greenfield site), displacement is not subtracted from gross impact. When an intervention leads to the unintended substitution of addressees for non-addressees, the effect should be considered as negative and subtrac­ ted from gross impacts (e. g. the retraining of unemployed males who replace female workers in local enterprises) If, however, these effects are expected or sought (e. g. the insertion of long-term unemployed which occurs at the expense of the short-term employed), the effect should not be subtracted from gross impacts. Displacement should not be put on an equal footing with deadweight since the latter is much more crucial to the estimation of the impact of Structural Fund interventions. When deadweight is substantial, the impact of an inter­ vention is correspondingly reduced. On the other hand, displacement only implies that impacts have occurred in one part of the assisted area at the expense of another part. The homogeneity of impacts has repercussions on how deadweight, displace­ ment and substitution are assessed. In the case of employment, impacts are homogenous and the estimation of deadweight and displacement focuses on a single evaluation criterion: the number of jobs created or maintained. Environmental impacts are, however, heterogeneous (e. g. air quality, water quality, bio-diversity, etc. ) and the net impact of an intervention may have to be estimated using a number of relevant indicators. Similarly, impacts of interventions on SME competitiveness (e. g. in terms of innovations, products, turnover, value added, etc. ) can be estimated using an array of indicators and the estimation of deadweight and displacement will depend on the impacts. The leverage effect, defined as the amount of private sector finance invested in an assisted project that can be directly attributed to Structural Fund co- financing of projects (see Box 7). This is another effect that should be estimated whenever an objective of interventions is to encourage the financial participation of the private sector. In addition to these effects, there are a number effects which, depending on the theme of the evaluation, will need to be taken into account. In the case, for example, of interventions in favour of employment, income multiplier effect (i. e. the employment created or maintained indirectly by the additional Technical considerations income generated by the direct employment effects of an intervention) and supplier effects (i. e. the additional employment created or maintained indi­ rectly in suppliers to assisted firms) may need to be estimated in order to have a more accurate picture of the employment impacts. It is important that these effects some times referred to as additional effects, are not confused with the notion of additionally as it is used by the European Commission. This concept is illustrated in Box 7. If the principle of additionality is fully respected, Structural Fund expenditure is fully addi­ tional to national government expenditure (lower left-hand figure). The opposite case is presented in the lower right-hand figure. This concept is a concern for financial monitoring at a national level rather than for programme evaluation. Box 7 - Additionality, deadweight and leverage effects Programme level SF expenditure Private expenditure expenditure Govt expenditure Govt expenditure 100% deadweight effect Leverage effect National level SF expenditure Private expenditure Private expenditure Private expenditure Govt expenditure Govt expenditure Govt expenditure SF expenditure Private expenditure Govt expenditure 100% additionality 0% additionality Il Thematic analysis of impacts on political priorities Structural Fund interventions are intended to promote economic and social cohesion and are complementary to the policy efforts of individual member states. To achieve this objective they selectively address the political priori ties of the European Union and the major preoccupations of European citizens. There are currently four political priorities which are specifically addressed by interventions: environmental protection, competitiveness, employment and the promotion of equality between men and women. The evaluation of political priorities can take place within a programme or across programmes (transnational or transregional thematic evaluation). In the first instance the objective is to provide conclusions on the extent to which a programme impacts on one or more policy priorities. The second scenario is that of an evaluation that is specifically centred on a given theme covering a number of different regions across a range of countries. A major objective of this type of evaluation is to highlight examples of best practice. The following chapters are structured around the key points that were presented in Part I, these key points being addressed and developed within the context of each of the four policy priorities. In addition to these points, a number of evaluation tools are presented that can be employed when an in- depth assessments of impacts is required. Evaluating the environmental impact of structural interventions ■ Why do environmental impacts need to be evaluated? • The regulatory framework For at least the past three decades growing international concern with the state of the environment has resulted in increasing provision for environmen­ tal matters in legislation and public policy. The European Union's response to this concern can be traced back to the Paris Declaration on the Environ­ ment in 1972. The Commission published its First Action Programme on the Environment in the following year and has commissioned research on envi­ ronmental impact assessment (EIA) since 1975. In 1977 work was begun on an EIA Directive which, after extensive consultation, was approved by Council in 1980. In 1985, Directive 85 / 337 / EC made it compulsory to assess the effects of a wide range of public and private projects, on the basis that "the best envi­ ronmental policy consists in preventing the creation of pollution or nuisances at source, rather than subsequently trying to counteract their effects". In 1992, the Treaty on European Union reflected an important shift towards a more active approach by requiring that environmental considerations be integrated into the formulation and implementation of all types of policy. A strategy for achieving sustainable development (i. e. development that meets the needs of the present without compromising the ability of future generations to meet their needs) in such an integrated fashion was spelt out in further detail in the Fifth Action Programme on the Environment which was adopted by Council in 1993. In this document it is stated that "The word 'sustainable' is intended to reflect a policy and strategy for continued economic and social development without detriment to the environment and the natural resources on which continued human activity and further deve­ lopment depend. " The economic development policies of the European Union thus increasin­ gly recognise the connection of development to the environment and ack­ nowledge that increasing the cohesion of the Union will demand a path that is sustainable in both economic and environmental terms. Since the reform of the Structural Funds in 1993, it has been required that explicit considera­ Transversal evaluation of impacts on intervention priorities tion be given to the environmental impact of all the activities assisted by the Funds: plans submitted by Member States seeking assistance under objec­ tives 1, 2 or 5b have since been expected to include "an appraisal of the environmental situation of the area concerned and an evaluation of the envi­ ronmental impact of the strategy and operations. "1. This requirement was underlined with the creation, in 1993, of the Cohesion Fund that devotes a substantial portion of its resources to projects with primarily environmental objectives. However, the primary objective of many interventions is not envi­ ronmental and there is often a tension between the type of activity which is typically deemed to be necessary to promote economic development, on the one hand, and environmental concerns, on the other. • The relationship between the environment and cohesion There is, of course, no immediate or automatic correlation between a region's level of economic development and the state of its environment. Both economically developed and underdeveloped regions can either have a healthy environment or experience severe environmental problems. In the past, controlling environmental impact was widely viewed as entailing costs and therefore as inevitably having a constraining effect on economic deve­ lopment. High growth, low environmental protection strategies are not sustainable over the medium to long-term. They need, therefore, to be replaced by more sustainable strategies that take advantage of a high quality environment in order to attract investment and use activities related to environmental impro­ vement as a vehicle for economic growth. Responding to relatively specific environmental problems may also offer short to medium-term employment opportunities and furnish necessary elements for wider economic develop­ ment strategies. The growth of "eco-industries" provides what is perhaps the most obvious example of the complementarity of environmental protection and economic development. These industries have developed to meet increasing demand from consumers for environmentally friendly products. They have also been a response to increased controls on pollution and the generally enhanced awareness of the importance of the environmental impact of human activity. 1. Council Regulation No 2081 /93, part III. Evaluating environmental impacts • The need for evaluation Since cohesion can develop only through activity that is sustainable In both economic and environmental terms, monitoring and evaluation need syste­ matically to take account of both economic and environmental factors. Indeed, the Commission is now obliged to respect environmental concerns in the design, monitoring and evaluation of structural interventions. In the case of structural interventions in lagging regions and rural zones, a sizeable number of measures have clear environmental objectives and the implications of supported activities for the environment are relatively direct, even if the measurement of the actual effects of such activity may be techni­ cally very difficult. In the case of interventions in other regions the environ­ mental consequences of supported activity are often less direct and more difficult to evaluate. While techniques for the evaluation of environmental impact have become increasingly sophisticated over the past few decades, for the most part they have concentrated on providing ex ante projections of likely impact so that informed choices might be made between alternative policies and projects. Ex ante environmental impact assessment is an increasingly significant and systematic element in the formulation and planning of structural interven­ tions. However, evaluation of the actual impact of these interventions (either individually or in total) is not yet widespread. Nevertheless, the Commission needs to ensure that methods are available for evaluating the actual environmental impact of the heterogeneous set of activities supported by the Structural Funds for three main reasons: • first, the overall orientation of the structural policies and the degree to which the efforts they devote to promoting cohesion through sustainable development have borne fruit, need to be checked by obtaining some indi­ cation of the environmental impact of structural policies as a whole; • second, whether the environmental priorities, which are often set out in supported programmes, have in fact been achieved needs to be verified; • finally, in relation to individual projects, project selection criteria related to the environment need to be validated by comparing predicted with actual outcomes. The EU's structural policies are delivered through fairly complex and hetero­ geneous programmes. Because an appreciation of the overall impact of Transversal evaluation of impacts on intervention priorities these policies is required, it is necessary to evaluate not only the cumulative consequences of particular projects on the environment with explicit envi­ ronmental objectives, but also the environmental implications of both projects and broader, macro­level policies which may have neither explicit environmental objectives or direct environmental impacts. Unfortunately, established methodologies for the evaluation of environmen­ tal impact have been developed almost exclusively for application to specific projects rather than to programmes or broader policies. It is true that, in recent years, strategic environmental assessment (SEA) has emerged as an approach to the evaluation of the environmental impact of broader policies. However, it appears that this approach is yet to be fully established technically and is, in any event, most useful in relation to ex ante evaluation. The approach presented here is by necessity less ambitious than a full eva­ luation of environmental impacts. This chapter is therefore oriented towards the evaluation of direct environmental impacts, focusing on those tech­ niques that in a simplified and operational form seem to be particularly well suited to ex post evaluation of environmental impacts. ■ Defining what needs to be evaluated • The environment and its influences. The concept of the environment can cover a vast range of topics that are inter­ related in complex ways. It is therefore important to clearly identify the main features of the environment and the most fundamental mechanisms shaping it. Natural elements may simply be consumed directly as resources. This, in itself, has an impact on the environment and can cause dramatic changes in eco­systems ­ changes which may be viewed as either positive or negative. In practice, human intervention in the environment always goes beyond simply consuming natural resources, to entail the establishment of infra­ structures and settlements, the use of energy, the transport of people and material, and the output of waste from each of these activities ­ all of which impact on the environment. All of these elements are interconnected and changes which occur may be either positive or negative. Evaluating environmental impacts Example: The environmental effects of agricultural activity Agricultural activity is highly varied in terms of both products produced and tech­ niques used. In the case of the use of intensive farming techniques, it can have negative environmental effects leading, for example, to a deterioration in the quality of soil, water and air, as well as the loss of habitats and bio-diversity. But agricultural activity can also have positive effects with crops acting as a sink for greenhouse gases, and with appropriate land management contributing to the conservation and enhancement of bio-diversity and the country landscape, and to the prevention of flooding and landslides. The basic determinants of environmental change are set out schematically in an adaptation of the "Driving Force-State-Response (DF-S-R) framework (See Box 8). This model represents the environment as comprising essen­ tially non-human elements that evolve in interaction with each other to form complex and delicately balanced eco-systems. It is one of a number of models that have been developed to explain the determinants of environ­ mental change (other more recent examples include the "Driving Force- State-Impact-Response" model of the European Environmental Agency and the OECD's adaptation of the DF-S-R framework to include agri-environ­ mental linkages and sustainable agriculture). Box 8 - An adaptation of the Driving Force-State-Response (DF-S-R) model of sustainable development Economy 1 transport energy industry agriculture trade tourism t_ State of the environment \ Wealth and welfare (economic) pressures or driving forces (economic) use of resources air water land wildlife natural resources responses (public and private) welfare (public action) responses Actors 4 administrations households enterprises international agencies Source: Adapted from Department of the Environment. (1993), Environmental Appraisal of Deve lopment Plans, A good practice guide, London: HMSO. Transversal evaluation of impacts on intervention priorities The environment can be broadly defined as the set of natural elements that surround an individual (human, animal or vegetal) or its species. Within the framework of the DF-S-R model this set of natural elements is comprised of air, water, land, wildlife and natural resources. This framework allows environmental change to be situated in the context of development in specific economic sectors to identify more accurately in terms of: • those driving forces, which can be defined as the elements causing changes (both positive and negative) in the state of the environment, these include: natural environmental process and factors, biophysical inputs and outputs, and socio-economic pressures; • the precise changes in the state, which reflects the changes in the condition of the environment, which are attributable to the various driving forces; • the policy and behavioural responses which are being made to those changes in the state of the environment. While this framework allows the issues involved to be reviewed in a clear and systematic way, and fosters a balanced consideration of environmental, economic and policy considerations, it does not help with the ultimate need to reconcile different types of criteria. This is particularly necessary in the matter of sustainable development since there is an element of tension between economic activity and environmental concerns. The very concept of sustainable development is based on the premise that economic growth should, at the very least, not entail unacceptable environmental cost. Fre­ quently it is difficult to define criteria which allow us to make judgements that are equally valid in relation to both economic and environmental considera­ tions. • Environmental impacts arising from the pursuit of non-environmental objectives Whilst the DF-S-R framework is unquestionably a useful tool for conceptua­ lising the determinants of environmental change, it is structured around a chain of causality which is circular in nature and therefore of limited use as a basis for evaluating the impacts of structural interventions on the environ­ ment. A more operational approach is to construct a typology of environ­ mental impacts, which combines socio-economic and environmental concerns and is generally relevant across the European Union. Evaluating environmental impacts For example, such a typology might consist of the following categories: • Water quality and resources • Atmosphere and material flows • Environmental management and innovation in enterprises • Land use and landscapes • Flora and fauna • Environmental behaviour of individuals The categories of this sort of typology are not intended to be mutually exclusive. For example, an action to increase public transport, which might have a direct impact on the "environmental behaviour of individuals", could also have the indirect impact on "atmosphere and material flows" through the diminished use of private vehicles leading to reduced exhaust emissions and demand for non-renewable energy resources. Likewise, impacts that fall within the category of "environmental management and innovation in enter­ prises" will tend to be upstream of impacts in other categories. However, it is unlikely that all of these levels of impact could be accurately traced. This type of approach allows the most direct impact to be specified while leaving open the potential to identify indirect impacts. It also allows specific themes to be analysed and reported on separately. Examples of effects (impacts and results) are classified according to the typology of impacts (see Box 9). Transversal evaluation of impacts on intervention priorities Box 9 - Detailed typology of environmental impacts of structural interventions Categories of impact Examples of impacts and results Water quality and resources • Increased proportion of Industrial and toxic waste water treated • Increased proportion of population connected to treatment facilities • Reduced use of nitrates in agriculture • Improvement in the quality of underground, river and marine water • Reduced leakage of treated water from mains system • Increased recycling of used water Atmosphere and material flows • Improved collection, treatment and recycling of solid industrial waste • Increased re-use, recovery and recycling of urban waste • Reduced size and number of landfill sites, and reduced pollution of underground water • Reduced Industrial emissions of CO2, sulphur and nitrogen oxides and unburned hydrocarbons Environmental management and innovation in enterprises • New patents in the field of environment related technologies • New enterprises specialising in environmental products and services • Increased investment in "clean technologies" Land use and landscapes • Treatment and reclamation of derelict industrial sites and abando­ ned urban areas • Increased number of areas designated for their landscape signifi­ cance • Natural areas protected from urban and industrial development, and development projects • Protection of areas and buildings of outstanding historical and archi­ tectural interest Flora and fauna • Reduced use of pesticides and herbicides in agriculture • Maintenance of bio-diversity • Cleaning up coastal areas, beach and rivers • Better management of timber and agricultural resources Environmental behaviour of individuals • Increased proportion of commuters using public transport • Increased proportion of tourists visiting off-peak • Reduction in use of private vehicles • Clarifying environmental objectives The object of evaluating the environmental impact of structural interventions is to obtain a measure of the effects of a mix of activities, all of which are intended to promote economic and social cohesion, but only some of which have environmental objectives. When environmental objectives are explicitly incorporated into an intervention they are obviously designed to result in a positive impact. However, other environmental impacts that result from inter­ ventions may be either positive or negative. Consequently, a complete envi­ ronmental impact assessment of these activities would need to take into account the full range of these effects. Evaluating environmental impacts Since the objectives of each component of a programme and the connec­ tions between different levels of a programme are rarely articulated as clearly and as explicitly as they might be, and since the environmental implications of particular interventions are frequently unclear, it is usually necessary to begin by clarifying the objectives. One approach to the clarification of the objectives of Structural Fund Interventions is to develop an objectives tree. Box 10 - Environmental objectives of Structural Fund Interventions presented in the form of a thematic objectives tree Global objectives Examples of specific objectives < Promote environmentally sustainable development Improved water quality - and resource management - Cleaner atmosphere and reduced material flows < - I • Increase the number of households and industrial plants connected to waste water facilities > • Improve the collection, treatment and recycling of industrial waste * - - -. 1 I 1 Better environmental management and innovation in enterprises • Increase environmentally oriented R &D • Increase energy efficiency of SMEs Better land use and landscape protection Better conditions for flora and fauna |. <-, I 1 • Make more industrial sites available in developed areas • Improve protection for areas designated for their landscape significance • Improve protection of marine areas and coastal zones More responsible behaviour in respect to the environment • Facilitating the use of public transport • Promote less environmentally damaging tourist activity By way of an illustration, the typology of potential impacts has been integra­ ted into an objectives tree specific to environmental impacts resulting from structural interventions. The global objective, which is to promote environ­ mentally sustainable development, is linked, via intermediate objectives, to examples of specific objectives relating to possible measures of a programme and other interventions. This example is largely based on explicit environmental objectives since the implicit objectives of a programme only become apparent when programme managers and decision-makers are interviewed. Relations between intermediate objectives are indicated by their relative position and the links between them (see Box 10). Transversal evaluation of impacts on intervention priorities The drawing up of an objectives tree facilitates: the specification of environ­ mental objectives of a programme, the clarification of relationships and any priorities that may exist between them, and better judgement of both the internal coherence of elements of a programme in relation to the environ­ ment and their external coherence with other public policies. • Gross and net environmental impacts The environment is subject to a complex combination of pressures arising from economic activity and the effects of individual and collective behaviour, as well as the effects of a range of public policy interventions. As a result, an observed environmental impact may not be wholly attributable to an intervention. It is therefore important to calculate the deadweight of intervention, so allowing its net effects to be estimated. A definition of deadweight in this context is the environmental change which may reflect the environmental objectives of an intervention but which would have occurred even without the intervention. Displacement in this context refers to the geographical shifting of negative environmental impacts rather than to their decline. This has emerged as an issue of concern in relation to the environment, for example, with regard to practices such as exporting toxic and radioactive waste from their sites of production to other locations that are no better equipped to cope with the presence of such substances. Example: Estimating the net Impacts of a town by-pass on the environment Assuming that after the completion of a town by-pass project, vehicle exhaust emissions in the town centre are estimated as having fallen by around 20% (the gross impact of the intervention). Yet, over the same period, the growth of private car use in the rest of the country has led to a 5% increase in exhaust emissions (counterfactual situation). However, although there is an obvious positive effect, there might be negative impacts on a number of residential areas on the outskirts of the town that are located near to the by-pass because of an increase in traffic levels as travel patterns change. The resulting increase in vehicle emissions, corresponding with the displacement effect of the inter­ vention, might be estimated at 10%. To estimate the gross impact of this hypothetical intervention the deadweight effect (counterfactual situation minus baseline) and the dis­ placement effect would have to be subtracted from the gross effect. It is also important to verify that none of the measures contained in a programme merely duplicate existing regional policies. A case of this would be if existing support mechanisms for general R&D projects were capable of Evaluating environmental impacts increasing environmentally oriented R&D. In that case, the environmental impact of environmentally oriented R&D may be positive, but it would not be entirely attributable to the measure being evaluated. A proportion of the impact observed would therefore have to be subtracted from gross impacts. ■ Examples of impacts by domain of intervention Three broad types of intervention are commonly supported by the Structural Funds: the provision of basic infrastructure, support for the productive envi­ ronment, and the development of human resources. All these types of inter­ vention may have possible environmental impacts and an example of each is dealt with separately below. • Basic infrastructure Support for the provision of basic infrastructure is usually intended to indi­ rectly promote cohesion by relaxing specific constraints on economic activity (e. g. by providing a high quality water supply which is suitable for specific production processes) or generally reducing production costs (e. g. by upgrading the road network). Infrastructure projects may have explicit environmental objectives and, whether or not this is the case, significant environmental impacts. In either case, evaluation of the economic impact of an infrastructure project needs to be complemented by an evaluation of its environmental impact if the sustainability of development based on this type of intervention is to be assessed. Transversal evaluation of impacts on intervention priorities Example: Light rail s c h e m es in t he United Kingdom The S o u th Yorkshire Supertram a nd Midland Metro were b o th f u n d ed under objective 2 in the United K i n g d o m. Both of these projects should result in the creation of new e m p l o y m e nt not only in the construction and operation of the new infrastructure but also b e c a u se of the Increased competitiveness of the region d ue to its improved links to transport corridors and decreased journey times. Both projects should also lead to reductions in noise a nd air pollution and enhanced values of investments in b o th resi­ dential a nd c o m m e r c i al properties. Outputs ΐ> Construction of a new light rail system Decreased volume of private vehicle use Effects Reduced noise levels > Reduced exhaust emissions Improved air quality Improved public transport links Reduced journey times Improved competitiveness Priority • Productive environment Various types of investment subsidy have traditionally been a major component of most regional development strategies, supporting the deve­ lopment of increased capacity for production, increased productivity, other types of direct support to individual enterprises (e. g. in relation to marketing, R&D or specifically environmentally friendly business practices), or indirect support by promoting complementary activities which may include environ­ mental services. Whether the environmental impact of this type of interven­ tion is positive or negative, any individual case will depend on the nature of the supported activity. Evaluating environmental impacts Example: Energy audits and advice for S M Es (United Kingdom) In an effort to e n h a n ce t he c o m p e t i t i v e n e ss of SMEs in the Knowsley area of Liverpool, the Powersave project supported experts from the local electricity supply c o m p a ny in visiting c o m p a n i e s, assessing their energy efficiency and p r o p o s i ng improvement plans. Putting these plans into action was c o - f u n d ed equally by the ERDF and the electricity company. For a total investment of 3. 7 MECU, cost savings of over 1 MECU were achieved annually and C 02 emissions were cut substantially. Outputs ^> [ Effects O Energy management plan for SMEs TJ Reduced fuel consumption Lower emissions of CO, Improved air quality Reduced demand for electric power Reduced production costs Improved competitiveness Priority • Human resources Interventions in support of human resource development are intended to strengthen and enhance the skills and abilities of beneficiaries and so improve their employability. The objective is also to make the local economy in which they work more competitive, owing to improved adaptability and productivity levels. To the extent that such interventions increase the envi­ ronmental awareness of beneficiaries or their capacity to engage in activities which are not detrimental to the environment or which have a positive envi­ ronmental impact, they increase the potential sustainability of the resulting development. A positive environmental impact is likely to result from the supported activity if the human resource development itself, or the employ­ ment that its beneficiaries eventually obtain, has an environmental objective. Transversal evaluation of impacts on intervention priorities Example: Developing recycling activities In Germany Assistance was provided from the Structural Funds to a training company in Geissen (Germany) which aimed to provide jobs and qualifications for the local unemployed, training for local employees and services with a positive environmental impact. The company was established in 1988. By 1996 it employed 370 people, was training 150 and had recycled thousands of domestic appliances (refrigerators, washing machines and dryers). This had also led to a significant reduction of CFC emissions and oil wastage. Outputs C> Effects ΐ> Provision of training places Recycling of domestic appliances Increased employment Increased skill levels in the area of recycling Reduced CFC emissions Reduced oil wastage Diversification of economy into eco-industries Priority It is generally difficult to say whether economic development will be accom­ panied by positive environmental spin-offs and consequently follow a sus­ tainable pattern over the longer term. However, it is important to bear in mind that a project that gives rise to negative environmental effects is not neces­ sarily a case of environmentally unsustainable economic development. Miti­ gation measures (i. e. measures that reduce the adverse effects of an activity on the environment) can be introduced which avoid, reduce or remedy the negative environmental impact of an intervention. These measures may succeed in stopping the deterioration of the environment or even lead to a net improvement, therefore achieving sustainability. Defining a nd selecting indicators • The need for and limitations of indicators Volume 2 of the MEANS Collection is devoted to the subject of indicators for monitoring and evaluating Structural Fund interventions. The above Volume includes an information sheet on indicators for the priority of the environment. Evaluating environmental impacts The overall environmental impacts of the EU's structural policies are the cumulative consequence of a wide range of different types of activity taking place in a similarly diverse set of local environmental contexts. To be in a position to obtain an accurate overview of the nature and scale of these impacts, access must be available to information that accurately reflects the most important aspects of change in the environment over time. In order for environmental change to be monitored, indicators should be able to show trends by incorporating a description of conditions which exist at a specified point in time (a baseline value) against which subsequent changes can be measured through monitoring. The number of indicators selected and defined should be limited enough to be both feasibly attainable and focused on the most relevant issues, but also comprehensive enough to reflect the most significant developments. For example, it is probable that many general levels of atmospheric or water contaminants would not be noticeably influen­ ced for a considerable time by even relatively substantial individual projects. In this case, for point sources of contamination (specific industries, major outlets) a simple measure of decreased output of specified pollutants from the source in question is more likely to be useful, while for diffuse sources (such as sectors of production - e. g. levels of use of artificial fertiliser, or broad patterns of behaviour - e. g. levels of private vehicle usage) a reasoned calcu­ lation of expected reductions in such outputs would be acceptable. • Recent efforts to develop indicators of sustainable development Considerable efforts to develop indicators of sustainable development on the international level have been made since 1992, notably by the UN, the OECD, and Eurostat. The United Nations Commission on Sustainable Deve­ lopment has produced individual methodological specifications for the defi­ nition, collection and interpretation of 132 different indicators. These indica­ tors are developed within the DF-S-R framework for four categories of information: economic, social, environmental and institutional. Eurostat produces a variety of publications which supply information on the environ­ ment (often prepared jointly with the OECD) and two on-line databases (New Cronos and Envstat). Eurostat has also been engaged in a pressure indices project which is intended to both establish a conceptual framework for measuring the impact of economic activity on the environment, and develop basic indicators for this purpose. The OECD has been engaged in a programme to develop policy-relevant environmental indicators and has working groups, for example, on environmental indicators for agriculture and for forestry. Finally, the European Environment Agency was established in Transversal evaluation of impacts on intervention priorities 1993 to produce information on the environment, including a comprehensive triennial state of the environment report on Europe, the first of which was published in 1995. The European Commission is also developing environmental indicators, including the establishment of a European System of Environmental Pressure Indices, comprising 50-100 physical pressure indicators which show trends for ten key "policy fields". Once these data series have been established, they will be to combine them into indices by attributing different weights to them, making the information they provide more manageable and better tailored to particular policy needs. Such indices are essentially proxies for more direct measurements of the impact of policies which are less readily available. For example, in the case of deforestation, an index would ideally not simply be an indicator of the rate of deforestation but something that combines this with an indicator of incentives for forest clearing. Similarly, instead of simply reporting the weight of herbicides used each year, a useful index would reflect the subject of eco-toxicity more directly. It is also possible to construct ad hoc regional level indicators for evaluating environmental impacts, as the following example shows. Example: The use of ad hoc indicators (France) The evaluation of the Burgundy objective 5b programme to support the development of rural zones required the construction of regional level indicators of environmental impact. The percentage of hedgerows destroyed was used as an environmental impact indicator of land regrouping operations. Furthermore, this measure was refined by categorising the destroyed hedgerows in terms of quality. In the area of public works, impacts were measured by the number of new beneficiaries of water treatment facilities, and in terms of the increase in the equivalent number of inhabitants having access to the facilities. The impact of waste management projects was measured by the increase In tonne / months of waste collected at disposal sites and the increase in tonne / months of selective waste collected. • Selecting indicators for a given programme Indicators are useful only if they reflect the changes which are to be evaluated. To an extent, the nature of these changes may be self-evident. Nonetheless, ensuring that they are adequately identified and interpreted requires some reflection. Regardless of whether the indicators of environmental impact which can be obtained relate to changes in the state of the environment itself or to changes in the forces acting on the environment, evaluators must be conscious of the variety of ways in which the environment may be influenced. Evaluating environmental impacts They may also be both the direct consequences of assisted activity (e. g. the damage caused by the construction of a major infrastructure project such as a motorway) and the indirect consequences of these or other actions (e. g. an action which may even have had a positive direct impact on the environment - the establishment of a country park - but which has ultimately led to negative impacts - the pressures on the environment resulting from visitors travelling to and using the park). Environmental impacts may also be of differing duration and be either discrete or cumulative. It is important to ensure that all of these types of environmental impact are considered in any evaluation. Once relevant changes in the environment have been identified, the interpre­ tation of how these changes were caused may be approached from two di­ rections. First, the outputs, results and direct impacts of the intervention in question can be monitored and reasonable inferences drawn regarding the impact that the outputs have on the overall state of the environment. Second, the state of the environment can be monitored and any changes noted may be attributed to any one or to a combination of a range of possible influences. Since the environment is open to such a variety of influences, it is often impractical to establish an empirical connection between programme outputs, on the one hand, and impacts which can be attributed to them, on the other (at least if impacts in terms of the state of the environment are to be measured). In such circumstances, the first is the most practical one to take, although it should be crosschecked when possible with the results of some efforts that follow the second approach. Indicators of the state of the environment have become increasingly sophis­ ticated in recent years. Examples of such indicators include: • an index of evapo-transpiration which provides a powerful indication of the state and character of the vegetation of a region; • an index of the total appropriation of net primary production of a region which provides an equally powerful indication of the degree to which vegeta­ tion is being exploited. Yet it is almost inconceivable that extensive practical use could be made of such indicators in evaluating the diverse and fragmented operations assisted by the Structural Funds. The strength of indicators of this kind in reflecting the environment is largely due to their independence from individual influences, but in terms of Structural Fund interventions this type of indicator Transversal evaluation of impacts on intervention priorities is too all embracing. Consequently, the impacts of supported projects would be lost amidst the numerous influences arising from other sources. For this reason, a "bottom-up" approach is appropriate for the selection of indicators for the ex-post evaluation of environmental impacts arising from Structural Fund Interventions. This approach consists of establishing a system of indicators, which allows the impacts of interventions to be tracked from the micro level towards the macro level. Evaluations of environmental impacts seek to answer two fundamentally different types of question requiring the use of different types of information. First, in relation to the components of an individual programme which had explicit environmental objectives, how successful were they in reaching those objectives? Second, in relation to all types of intervention, what is the potential environmental impact at the programme level or in terms of groups of programmes (e. g. those aimed at a particular global objective of the struc­ tural policies or those in a particular Member State)? In the first instance, it is often difficult to definitively attribute changes in the overall state of even particular aspects of the environment, to specific projects. Reasonable assumptions have to be made on the likely impact of assisted activity on the basis of information that does not precisely reflect the state of the environment. In some cases this is perfectly satisfactory (e. g. in the case of clear reductions of discharges from point sources of water or air pollution). However, the more indirect the relationship between interven­ tions and changes in the state of the environment (e. g. in the case of human resource measures for training in environmentally relevant skills), the more difficult it is to develop a convincing measure of impact. This problem is less taxing than it might be due to the fact that the environ­ mental objectives of a structural intervention are themselves often expressed in terms of change in driving force variables. These objectives make assumptions about the relationships of such variables to the state of the environment. Provided such assumptions are credible, it is best to focus the evaluation of the environmental impact of the programme in question on the analysis of the driving forces concerned. In the second instance, it might be expected that because the scale of the overall effort is larger, impact is easier to identify in terms of state variables. This may be the case in some instances but usually information on the Evaluating environmental impacts impacts of the overall effort is itself also available, at best, only in terms of driving-force variables. Moreover, evaluation of impact at this level is complicated by the need to aggregate a variety of different measures of impact which, even if they refer to the same variable, may have been expressed for good reason on the programme level in ways which are context-specific. For this reason, when evaluating environmental impact on the multi-programme / policy level, it is necessary to refer more systematically to established data-series on state variables, even if it is impossible to establish a direct link between these and the information which is available on the actual impacts of interventions. By way of an example, a system of indicators which could be used to track the production of impacts, both environmental and socio-economic arising from the construction of waste water treatment plants is presented in Box 11. Box 11 - Example of a system of indicators for the construction of waste-water treatment plants Programme Indicators (Related to the intervention and Its effects) Context Indicators (Related to the assisted area) Output Indicators Progress Capacity Result Indicators Compliance with project duration Rate of completion Number of new waste water treatment plants constructed Maximum annual throughput (m1) Treatment in relation % of waste-water undergoing to demand and pro- cessing standards ! primary treatment % of waste water undergoing secondary treatment Impact indicators Consumption Indirect economic effect Volume of waste-water treated Number of households / businesses connected to waste-water treatment facilities by programme after one year % of waste water treated % of households / businesses connected to waste water treatment facilities Number of economic units (firms, farms, etc. ) declaring that the new water supply system has released a major constraint for their development Environment % of waste water samples from point sources which show a specified level of decline of Identified pollutants % of km of rivers monitored on which pollution overcome a specified standard Source: Volume 2 of the MEANS Collection Transversal evaluation of impacts on intervention priorities Although the "top­down" approach to measuring environmental impacts is dominated by the use of quantitative data (e. g. the indices constructed by Eurostat), a "bottom­up" approach can be based on qualitative data. Fur­ thermore, this qualitative data can later be transformed into a quantitative indicator, should this be necessary. Example: Constructing an environmental impact indicator on qualitative data A French evaluation team had to evaluate the environmental impact of a motorway. Their method consisted of sending a group of people to photograph what they consi­ dered to be landscape blackspots in the area of the motorway before corrective measures had been undertaken. The same photographic survey was then carried out after the corrective measures had been undertaken. The change in the number of environmental blackspots was then recorded. To summarise, indicators will have to be selected which are neither simply the standard type of socio­economic indicators traditionally used in the eva­ luation of structural policies, nor the specialist technical indicators which are being developed to monitor change in the state of the environment. These indicators will tend to resemble the traditional systems of indicators used for the evaluation of structural interventions in relation to resource, output and result indicators. However, the impact indicators will be largely made up of indicators of changing driving forces on (rather than the changing state of) the environment. ■ Principal sources of information Each programming document should contain the following information on the environment: • a description of the main environmental characteristics of the geographical area covered by the programme; • an identification of any sub­areas of particular environmental significance; • the nature, cause and location of any significant pollution or other environ­ mental driving force problems; • a description of existing national / regional environmental policies; • information on national / regional environmental authorities and procedures; • a projection of the expected environmental impact of the programme as a whole and of each constituent priority / measure; • details of any steps which are proposed to mitigate any negative environ­ mental impacts of the programme or its components. Evaluating environmental impacts Although all of these details are not always found in a programming document, these documents are still the obvious starting point for obtaining information. Programme monitoring systems should provide data on resource expendi­ ture and outputs needed to answer many evaluation questions, but are unlikely to provide data on the evolution of the environment itself in the region concerned. However, in cases where the preparation of an interven­ tion has shown that gaps exist in the availability of specific context data that are of particular importance in evaluating its impacts, provision may be made to obtain this information within the monitoring system. Established statistical time series of environmental data provide an increa­ singly rich source of information that can be used to complement the infor­ mation provided in programming documents. Information from programming documents and monitoring systems can be expected to provide very specific micro­level information on the allocation of resources and their impacts in terms of driving forces on the environment. More established data series tend to concentrate on macro­level state variables. One of the tasks of evaluation is to try to draw connections between these two levels of information, and in carrying out this task it may also need to draw on information from previous research and evaluations or engage in original data collection and / or production. ■ S o me examples of current practice in the European Union While a more detailed treatment of evaluation tools is provided in Volume 3 of the MEANS Collection, a number of examples in­depth analyses of envi­ ronmental impacts are presented below. A variety of tools has been used in the identification and analysis of the envi­ ronmental impact of public policy. Selecting the most appropriate evaluation method or technique to use in any instance depends on the particular eva­ luation requirements that prevail. It is important to note, however, that each technique has its own strengths and limitations and, even within the European Union, the choice made is likely to be influenced by relevant national legislation and established national practices in relation to: project planning; the series of pre­feasibility, feasibility and technical studies which are typically required; and, lastly, established implementation, monitoring and audit procedures. Transversal evaluation of impacts on intervention priorities An assessment of the impact of interventions on the environment can be undertaken using the "step-by-step" approach presented in Part IV of this Volume. The choice of tools used in this instance would focus on minimal case studies, individual interviews and focus groups. On the other hand, for an in-depth evaluation of environmental impact analysis, other evaluation tools have to be employed. • Strategic environmental assessment of programmes in the Netherlands Strategic environmental assessment (SEA) refers to the environmental assessment of programmes and policies as opposed to individual projects (the latter is the traditional function of environmental impact assessment - EIA). SEA developed as a result of the realisation that EIA may occur too late in the planning process to allow adequate consideration to be given to alter­ natives. SEA is a formalisation of the process in which the other techniques mentioned in this section are applied, and an SEA system is expected to include formal guidelines on the use of scenario techniques, screening and scoping, case studies, etc. SEA takes place in the context of a tiered forward planning process where policies at the international / national level are seen as shaping programmes at the regional level and projects at the local level. Within the EU this approach is adopted with increasing frequency in relation to waste manage­ ment, land-use and energy policies. Within the EU, the Netherlands is the country where the practice of strategic environmental assessment (SEA) is most developed. The Dutch experience in this area dates back to the early 1980s and SEA is now legally required for various activities including electricity and water supply, nature conservation and outdoor recreation, and provincial waste management proposals. A National Programme on Waste Management has been in place in the Netherlands since 1992. This programme is revised every three years and an integrated set of ElAs as well as a SEA have been implemented on a voluntary basis by the Waste Management Council - a partnership of local, regional and national authorities. The approach works on three levels: • Decisions are made on the national level concerning the waste treatment options / technologies that might be used. These decisions are subject to a SEA before a selection is made. • Decisions are made on the regional level regarding the possible location of treatment facilities; these are also subject to a SEA. Evaluating environmental impacts • Finally, individual project alternatives are selected for each of the chosen locations; these are subject to ElAs. The SEA considered two alternative scenarios. First, a situation in which the objectives of the National Environmental Policy Plan on recycling and the production of waste were met. Second, a situation in which these broad policy objectives might not be attained but individual programmes and measures were still expected to be effective in reaching their more limited objectives. Based on this analysis, a strategy was formulated for the medium-term development of waste disposal facilities. Regular meetings with provincial authorities (responsible for implementing the policy) and interest groups were an integral part of the development of the SEA. The policy was then implemented by each of the 12 provincial authorities, with each one developing four-year programmes. In each case global objectives were set (e. g. 400 factories should have a waste prevention plan by the end of the programme), needs calculated, the potential for improvement in relation to specific indicators estimated and project selection criteria established. Within the provincial programmes project selection is usually carried out on the basis of ElAs. In view of the fact the EIA method is more established, it is not surprising that work on the project level has proved to be more evidently methodical, with a clear focus on assessing alternative processing tech­ niques, and a definite choice of indicators. The relationship of output from this project level activity to the SEA is not clear-cut and it seems that the latter exercise has an inherently poorer quality of information. With regard to intermediate or ex-post evaluation, even though Dutch legislation is more demanding in this respect than that of many other countries, there is only advice (rather than a requirement) that this be implemented. Even if there is no obligation to adopt technically sophisticated evaluation techniques within SEAs, experience has shown that the formalisation of the process itself has brought a number of benefits to the design and implementation of policies with an environmental impact. The following advantages of the formal inte­ grated approach offered by SEAs have been identified by administrators: • environmental issues are given a greater degree of influence on general policy formulation; • decision makers routinely use those results of the SEA process which are clearly well designed; • although test case experiences may not - strictly speaking - be generali­ sed, an impression of their relevance is more readily obtained in the context of a SEA and their results are more likely to influence decision-making. Transversal evaluation of impacts on intervention priorities Further potential benefits of adopting a SEA approach are: • it encourages consideration of alternatives which might otherwise be ignored • it allows more effective analysis of the cumulative effects of projects • it facilitates both the planning and analysis of synergistic effects • it is better able to evaluate ancillary / secondary effects of actions • it is capable of taking delayed impacts into account • it permits the development of standard, cross-measure mitigation provision • it can enhance both communication between those responsible and the involvement of the public • it provides a framework in which cumulative and synergistic environmental impacts can betaken into account in a systematic fashion. • The screening and scoping of a programme in the UK The concept of screening refers to two different types of process depending on the level at which it is applied. Within a measure, screening involves the selection of projects for assistance through the application of a standard set of project selection criteria. An example of this process is supplied below. At the measure level itself, screening refers to the initial examination of measures to determine the degree of environmental impact assessment which is required before approval is given for their implementation. The process of scoping, on the other hand, is concerned with determining which potential impacts of alternative projects deserve consideration in an EIA. In practice, this is a considerably more complex process as there may not be agreement among interested parties as to precisely which impacts are relevant. Under UK legislation, specific types of physical project are either classified as being subject to mandatory environmental impact assessment (typically very large capital projects with a clear and direct environmental impact) or checked against a number of criteria (such as the location, scale, proposed method of dealing with waste, etc. ) of the individual project to judge whether a full EIA is needed. Scoping is carried out in voluntary discussions between project developers and local planning authorities prior to the submission of environmental statements on the proposed projects. The region of Merseyside in England is the subject of a programme of assisted activity for the period 1994-1999 under objective 1 of the structural Evaluating environmental impacts policies. Each project seeking assistance under this programme is screened in a two-stage process: • an initial stage which verifies that the proposed project meets UK statutory controls; • a second stage which scores proposals against environmental criteria which have been separately established for each of the programme's priori­ ties and measures and also takes account of the precise location of the proposed project. A summarised example of an environmental screening form for a particular measure of the Merseyside programme is provided in Box 12. This approach is of particular interest as an example of combining economic objectives with the other objectives pursued under a specific programme assisted by the Structural Funds. Box 12 - Example of an environmental screening form Priority 1: Measure 1: Quality sites and premises for industrial and commercial investors. Objective: To ensure an adequate provision of quality industrial and commercial sites and premises encourage indigenous growth and Increased employment opportunities. to Project: Reference: Yes N/A No 1. If the development Is on a brownfield site, will It reduce the area of derelict land or return buildings to productive use? 2. Does the development respect designations regarding the natural environment or built heritage? 3. Does the development incorporate good practice for energy conservation, water recycling or waste reduction or involve innovative environmental measures? 4. Will the site have an environmental management plan or are measures In place for post-development environmental management? 5. Have any measures been taken to ensure that the site is easily accessible by public transport? 6. Is the development designed to Improve the local built environment, for example by minimising the effects of crime, vandalism or litter? 7. Does the development project contain proposals for environmental Improvements such as landscaping? 1 1 1 1 1 0. 5 0. 5 0. 5 0. 5 0. 5 0. 75 0. 38 0. 75 0. 38 Total score: If the project Is In a priority area multiply this score by 1. 5 and round to the nearest 1. 0 0 0 0 0 0 0 In those sub-areas of the Merseyside region which are judged to experience particularly severe environmental problems or where environmental impro­ vement could be expected to have an unusually strong impact on develop­ ment, the total score of environmental criteria is positively weighted in Transversal evaluation of impacts on intervention priorities relation to other project selection criteria. These particular sub-areas were themselves defined according to a number of criteria, including: the visibility and reputation of the site in question, its proximity to strategic transportation corridors or locations regarded as assets, and whether it already qualified for assistance from other sources of funding. Council Directive 85 / 337 / EEC specifies all projects for which assessment is mandatory and others to which particular thresholds and criteria apply. However, in the context of programmes supported by the Structural Funds, Member States are also required to ensure that any individual operation is compatible with EU environmental policy and legislation. Some form of screening is therefore required of all the components of any prograrhme, but much depends on the quality of the specific criteria developed and the design of the process adopted in each instance. Screening is obviously necessary at the ex ante phase and should at least be confirmed during subsequent phases of evaluation. It may also be useful for ex-post evaluations to adopt a screening technique to focus analysis on those measures likely to produce the greatest impact. • Case studies of projects in Scotland It is often useful to select individual examples of a phenomenon for in-depth study, for example: • when exploratory fieldwork is required due to the absence of adequate information; • to provide a detailed analysis of a particular process of environmental impact; • to test a hypothesis or assertion about a particular project. Case studies may be individual or multiple and in the latter case may involve sophisticated techniques to ensure the replication of design in different contexts. The approach has four stages: design and selection, data collec­ tion, data analysis and report preparation. Selection of the case(s) for study is particularly important for the relevance of findings. Cases may be selected on the basis of being either typical or representative of a situation which is of interest for other reasons (e. g. best / worst cases, particular circumstances). Data collection may involve the use of surveys and interviews. In an attempt to develop a useful method for appraising environmental projects intended to restore or protect specific sites, the Scottish Office engaged evaluators to review methods of valuing environmental impacts. Evaluating environmental impacts On the basis of the first stage of the study and in view of the type of projects being considered, it was decided that the contingent valuation method (the various impacts are valued by the beneficiaries and losers through question­ naires), in particular, warranted further investigation. The second stage of the work involved applying this method to seven projects. The cases studied include the improvement of an old industrial site, the restoration of an urban area, the protection of a mountain area and woods, and support for an industrial heritage centre. In each case members of the public were intervie­ wed or surveyed. The results of this case study confirmed the value of the technique examined and gave a clearer indication of the circumstances in which it could usefully be applied. Multiple case studies can be used to analyse the effects of interventions in a number of contexts across the programme area. Provided adequate provi­ sions were made for design replication, they could be used to better identify impacts by comparing situations in which a measure was applied to those in which it was not. Another subject, which appears to offer particular scope for the application of this technique, is the relationship between changes in pressure variables and state variables. Coefficients might be developed from case studies that would allow calculations of impact on the state of the environment to be made on the basis of indicators of change in pressures. While the nature of effects can be determined inductively by using the case- study technique, this technique cannot, strictly speaking, demonstrate causality. However, since the evaluation of environmental impact generally needs to make an inductive leap of faith from knowledge about change in driving force variables to related changes in environmental state variables, a case-study approach is at no disadvantage in this respect. • Combining cost-benefit & multi-criteria analysis to assess projects in Ireland Cost-benefit analysis attempts to measure in monetary terms all of the costs and benefits of a project over the course of its lifetime. It is commonly used at the ex ante stage of project evaluation, to choose between a number of alternative projects and to judge for which, if any, the calculated economic benefits exceed the economic costs involved. In a more sophisticated version of cost-benefit analysis, instead of attemp­ ting to express all benefits in terms of a single monetary value, a number of explicit evaluation criteria are established which can be weighted by Transversal evaluation of impacts on intervention priorities decision makers to allow for a ranking of alternatives. This approach is known as multi-criteria analysis (MCA) and, in relation to our present concerns, offers the advantage of permitting systematic account to be taken in the decision-making process of both environmental costs and economic costs, rather than attempting to translate all environmental criteria into monetary terms. In Ireland, the Department of the Environment engaged consultants to undertake evaluations both to meet EU requirements for the selection of specific environmental infrastructure projects (co-funded in this case by the Cohesion Fund) and to establish a methodology for future decision-making. Traditional cost-benefit analysis was rejected by the consultants who recognised that the quantification of benefits in monetary terms posed too many problems in this case (water supply, wastewater treatment and water quality management projects). One potential weakness of adopting a multi-criteria approach is that the weightings used are essentially subjective and scores are sometimes arbitrary. For this reason, the approach adopted combined the techniques for identifying criteria used in MCA, with a common denominator of monetary values derived from relevant studies. In each case the basis of the weightings used was explained. This approach was applied with variations for each of the three types of project to be evaluated. In the case of water supply projects, the approach had four components: • identification of all costs associated with each alternative project; • quantification of all direct and indirect benefits; • ascribing values to benefits for each project; • calculation of benefit / costs ratios. Information was collected on pre- and post-project demand levels broken down by sector, population served and the design demand and population for each project. Standard operating costs were assumed to apply to each alternative and three types of benefit were identified: • quality improvements • extension of supply to new consumers • wastage elimination and control. Each of these three benefits was assigned a specific unit value and a weighting derived from analysis of relevant studies. The basis of these values was identified and both the reasons for using them and the limits to their applicability were explained. Competing projects were then analysed using a number of different scenarios of likely demand evolution and a benefit / cost ratio was calculated for each one. A ranking of the projects was Evaluating environmental impacts made and the results of this ranking were clearly explained so that the impli­ cations for decision­makers were obvious. All of the variants on this approach are essentially tools for decision­making in the selection of alternative projects. Much depends on the professiona­ lism of evaluators in clearly indicating either the assumptions involved in establishing single indices for ranking projects or (in the case of MCA) the need for decision­makers to themselves arbitrate between criteria. To the extent that these techniques are applied rigorously, they can be a useful means not only of appraising the relative strengths and weaknesses of alternative projects but also of providing a synthetic score for their sustai­ nability which can then be checked against actual performance. ■ Summary The quality of the environment and how it can be maintained in the face of economic pressure is an issue that has risen rapidly to the top of the political agenda. At the same time, there has been an increasing acknowledgement that economic development does not have to involve a trade­off in environ­ mental terms and that the two can, in fact, be mutually reinforcing. However, ensuring that economic development and cohesion are sustainable requires that both economic and environmental factors are taken into account when evaluating structural interventions. For evaluation, this means that interme­ diate and ex post evaluation of environmental impacts of projects and pro­ grammes need to be undertaken, in addition to the more common practice of assessing projects for their environmental effects. Although the standard DF­S­R framework may be a useful tool in other cir­ cumstances, it is of little use for the evaluation of the effects of Structural Fund interventions on the environment. A "bottom­up" approach is therefore proposed which, at the impact level, focuses on driving force variables, combining both socio­economic and environmental concerns. The predominance of ex ante project assessment means that there are few examples of good practice in the use of tools in intermediate and ex post evaluation. Nearly all the examples presented are thus of tools that have been used in circumstances of ex ante project assessment, but which can generally be adapted to other evaluation situations. Transversal evaluation of impacts on intervention priorities ■ For m o re information see. Department of the Environment. (1993), Environmental Appraisal of Develop­ ment Plans, A good practice guide, London: HMSO. Ecotec Research and Consulting Ltd. (1997), Encouraging Sustainable Development through Objective 2 Programmes: Guidance for Desk Officers, Note of the European Commission (DGXVI). European Commission (1998), A Handbook on Environmental Assessment of Regional Development Plans and EU Structural Funds Programmes, European Commission, DG XVI. European Commission (1999), The Thematic Evaluation of the Impact of the Structural Funds on the Environment, European Commission, DG XVI. European Environment Agency. (1996), Guidelines for Data Collection for The Dobris+3 Report, Technical Report, Copenhagen: EEA. European Environment Agency. (1995), Environment in the European Union, Report for the Fifth Environmental Action Programme, Copenhagen: EEA. Fehily, Timoney, Weston. (1995), The Economic Appraisal of Environmental Projects Supported by the EU Cohesion Fund, Report for the Department of the Environment, Dublin. OCDE. (1996), La Cohérence dans l'Evaluation Environnementale, Orienta­ tions générales pratiques pour les projets de coopération pour le développe­ ment, Paris: OCDE. OECD. (1997), Environmental Indicators for Agriculture, Paris: Publications Service, OECD. Sadler B. (1996), Environmental Assessment in a Changing World : Evalua­ ting Practice to Improve Performance, International Study of the Effective­ ness of Environmental Assessment, Canadian Environmental Assessment Agency, Minister of Supply and Services Canada. Statistical Office of the European Community. (1997), Indicators of Sustai­ nable Development, Luxembourg: Office for Official Publications of the European Communities. Wood C. (1995), Environmental Impact Assessment, A comparative review, Harlow: Addison Wesley Longman Ltd. Evaluating the impact of structural interventions on the competitiveness of Small and Medium­Sized Enterprises SMES ■ Why evaluate impacts on Small and Medium­Sized Enterprises? Small and Medium­Sized Enterprises (SMEs) make an important contribu­ tion to job creation, competitiveness and growth. Net job creation, particu­ larly in enterprises with fewer than 100 employees, more than compensated for job losses in large enterprises during the period 1988 to 1995. Despite this apparent dynamism, employment created by new enterprises tends to be somewhat unstable, with only half of new businesses created still in existence after five years. Box 13 ­ Definition of SMEs The European Commission has adopted a Recommendation concerning a standard definition of SMEs: Medium­sized enterprises have less than 250 employees, an annual turnover of less than 40 MECU or balance sheet assets of less than 27 MECU and must be independent, which means having less than 25% of their capital owned by a large enterprise. Small enterprises have to have less than 50 employees, a turnover of less than 7 MECU or assets of less than 5 MECU and must be independent. Micro­enterprises are defined as firms with less than 10 employees (The recommendation on the definition of SMEs of 3 April 1996, can be found in the JOL 107 of 30. 4. 1996). The Madrid SME Policy Paper2 stresses that European SMEs' full potential for growth and employment creation has not yet been properly realised for a number of reasons. Firstly, it has been argued that the complex legal, fiscal and administrative environment faced by European enterprises has a relati­ vely higher impact on SMEs than on large companies. Secondly, SMEs often have problems accessing bank loans at reasonable rates or raising equity on the markets. In particular, small enterprises have difficulties in gaining access to parts of the market due to a lack of comprehensive market intelli­ gence or problems in identifying suitable partners. Others have argued that the business environment in Europe (in particular, the lack of sufficient risk capital, the existence of significant regulatory barriers and an aversion to risk taking) is not as conducive to the development of SMEs and, in particular, of 2. Commission Communication of 29. 11. 1995 entitled Small and Medium Sized Enterprises ­ A dynamic source of employment, growth and competitiveness in the European Union, Document CSE (95)2087. Transversal evaluation of impacts on intervention priorities New Technology Based Firms (a restrictive definition of New Technology Based Firms - NTBFs - is that of firms active in the developing industrial sectors based on emerging technological paradigms such as bio-techno­ logy or laser technology). For the purpose of this document, the wider term of high-tech firm covering all enterprises active in high-tech sectors or producing high-tech products will be used (a typology of sectors and products has been developed by the OECD). Where SMEs are not techno­ logy intensive, better use needs to be made of readily available technologies in order to remain competitive. However, very few SMEs search patent data banks in order to find out what technology is available "off-the-shelf". In less favoured rural areas SMEs account for a relatively high share of employees in the local economy but also face particular problems. Whilst difficulties vary between regions, the quality of infrastructure and the distance from main markets can sometimes be a problem, as can poor access to information, poorly developed local industrial base and a lack of suitably qualified workers. Although these factors pose problems for the development of SMEs in general, in some regions self-employment and entrepreneurship are values that are recognised and encouraged. Thus, some less favoured rural areas belong to the most dynamic regions, in terms of new and small business development, and generate more new jobs than the national average. • The Community context Recent European Councils have given a positive political signal towards SMEs, notably in Amsterdam (June 1997), Luxembourg (November 1997) and in Cardiff (June 1998). As a result of the Amsterdam Council, the BEST task force was created and presented its report to the Commission in 1998. This report includes a number of recommendations to improve the business environment, some of which should help SMEs to grasp the opportunities offered by Community programmes. The Luxembourg Special Employment European Council recognised the promotion of entrepreneurship as a common guideline for Member States' employment policies within the Union. As a follow up, the Commission has adopted a Communication entitled "Fostering Entrepreneurship in Europe - Priorities for the Future" setting out a number of priority issues both at European and national levels. Moreover, the conclusions of the Job Summit gave a positive signal towards SMEs by sup­ porting the further development of financial instruments and cross border part­ nerships. The Cardiff European Council has asked the Commission to present a report in Vienna on the follow up of the BEST recommendations. Evaluating impacts on the competitiveness of SMEs Since SMEs play a primary role in balanced local and regional development, and are a driving force behind effective adaptation to structural change and innovation, the European Commission has recommended that, within the overall framework of interventions in industry and services, it was necessary to take the needs of SMEs systematically into account in the implementation of programmes. For this purpose, the Commission has developed the Integrated Programme, a global framework for co-ordination of all activities for SMEs. This includes the Third Multi-Annual Programme for SMEs (1997-2000), the contributions that other policies make to the development of SMEs, and the Concerted Actions which have the goal of promoting the co-ordination of Member States' SME policies through the exchange of best practices. To accomplish this aim, the following five priorities are set out in the Integra­ ted Programme: • simplify and improve the administrative and regulatory business environ­ ment; • improve the financial environment for enterprises; • help SMEs to Européanise and internationalise their strategies, in particu­ lar through better information and services; • enhance SMEs' competitiveness and improve their access to research, innovation and training; • promote entrepreneurship and support special target groups such as women and young entrepreneurs. Structural Funds contribute to achieving the objectives of SME policy through their support for the creation and development of SMEs in the various operational programmes. The European Investment Bank and the European Investment Fund are also involved in providing support for SMEs. Although the main objectives of Structural Fund interventions in favour of SMEs are similar across the EU, there are differences in the specific objec­ tives from one region to another. Thus, in objective 1 regions there is an emphasis on strengthening the support structure for SMEs. In objective 2 regions where large-scale traditional industries have declined, measures focus on promoting entrepreneurship and creating alternative job opportuni­ ties. In objective 5b regions the emphasis is on supporting the diversification and modernisation of rural SMEs and in objective 6 areas the priority is to help SMEs in overcoming problems caused by their distance from major population centres. Transversal evaluation of impacts on intervention priorities The development of the potential of European SMEs in general, and in relation to the role they play in social and economic cohesion in particular, represents a major strategic aim of the European Union. The level of support for SMEs in terms of resources allocated from the Structural Funds for the period 1994­1999 reflects the importance that the Commission attaches to small and medium­sized firms. For instance, on average 10% of ERDF resources are devoted specifically to measures dealing with the promotion of SMEs and the improvement of the framework conditions in which they operate (Framework conditions correspond with the business environment in the widest sense, incorporating: the overall macroeconomic conditions, tax system, regulatory environment, bankruptcy laws, competition policy, etc. ). This is almost double the level of support provided in the period 1989 ­1993. The purpose of evaluating the impacts of Structural Fund interventions with respect to the priority of SME competitiveness is defined by the European Commission as being to "provide a thorough and systematic analysis of the contribution and impact of Structural Funds' support to SMEs in the EU Member States and regions". Thus, evaluation has a major role to play in assessing the extent to which policy objectives have been or are being achieved, in providing insights into the impacts of Structural Funds on SMEs, in guiding the process of policy making and, consequently, in enhancing the effectiveness of future programmes in this domain. ■ Defining w h at n e e ds to be evaluated Evaluations of Structural Fund interventions in favour of SMEs need to look at the strategies underpinning the implementation of programmes (i. e. how well­founded the strategies are in relation to the strengths and weaknesses of the assisted regions, to other public policies related to SMEs and to current ideas on SME support). They also need to examine the implementa­ tion of measures in favour of SMEs in order to assess them in terms of effi­ ciency and effectiveness. Impacts of measures not specifically targeted at SMEs should also be assessed to see if interventions are benefiting SMEs while meeting their stated objectives (i. e. increasing employment or growth). • Guidelines and key issues in SME evaluations The effectiveness of Structural Fund interventions in favour of SMEs depends as much on how efficiently support is delivered as on their design. For this reason it is important to examine the implementation of pro­ grammes. The following key issues can be used as guidelines: Evaluating impacts on the competitiveness of SMEs • How are support measures being delivered? Has the fragmentation of SME support structures and schemes that exists in some countries been reduced by the introduction of innovations such as "one stop shops" (a function performed, for example, by Chambers of Commerce in Germany and Business Links in the UK)? Are addressees less confused about support schemes? Who manages SME programmes and to what extent are they implemented through partnerships between public and private bodies? In regions where institutional structures are relatively weak, to what extent have SME support programme delivery mechanisms been strengthened through institutional restructuring, capacity building or the involvement of private agencies? • How are the different types of SME programmes themselves appraised? How is the implementation of measures monitored and evaluated by different national and regional authorities? How will the results of the evalua­ tion be taken into account in the formulation of future policy developments or the reorientation of existing programmes? There are a number of key issues related to the impact on regional develop­ ment of Structural Fund interventions in favour of SMEs: • Interventions aim at both encouraging start-up of new businesses (e. g. by providing seed capital) and supporting existing SMEs. Issues of survival and performance of assisted enterprises are therefore both relevant. • Interventions also aim at promoting the exchange of information and the diffusion of best practices between SMEs situated in different regions. The extent of transborder networking is therefore an important issue for evaluation. • To a certain extent, the creation of employment and the creation of wealth at the micro-level involve a trade-off between these objectives. It may therefore be important to examine to what extent assisted enterprises reinvest profits to assure their survival over the medium and long-term, and the potential of these investment to engender job creation. Furthermore, when employment creation or maintenance can be attributed to the contribution of Structural Fund interventions, the cost should, whenever possible, be estimated. • Many small firms produce a narrow range of products and services. This means that when diversification is the objective of an intervention, it is the emergence of new firms specialising in new market niches, rather than the reo­ rientation of existing ones, that will tend to be the main motor of this process. • SMEs may also have a very limited client base, and in some cases they can find themselves in a position where they have only one buyer of their products or services. In this situation close relations with larger firms can have a positive effect as subcontractors adopt technological and perfor- Transversal evaluation of impacts on intervention priorities mance standards of the contracting firm, or benefit from technology transfer, technical training and assistance. On the other hand, the competitive position of such firms is likely to be weak, and the economic fabric of the region will, as a consequence, be fragile. The place of assisted enterprises in supply chains and clusters may in certain cases represent a key evaluation issue. When comparing different programmes at the national or regional level, top- down techniques can be employed to gather relevant contextual informa­ tion. On the other hand, the use of bottom-up techniques allows data collec­ tion about programme outputs and results, such as the number of SMEs assisted, the number of new businesses assisted, and the efficiency and effectiveness of the implementation of interventions. The estimation of the impacts of interventions at the project and measure level also lends itself to the bottom-up approach. It is important, when looking at measures, to distinguish between those which are directly targeted at SMEs and which are easily identified in Opera­ tional Programmes, and those from which SMEs benefit only indirectly and which are therefore much more difficult to trace. Finally, it is essential to distinguish between addressees in terms of their size by using the classification of SMEs, including micro-enterprises, established by the Community. • Clarifying the objectives of SME interventions An early step in the evaluation of SME interventions is to develop a hierarchy of objectives in order to be able to examine the strategic aims of the Structu­ ral Fund programmes at various levels. This can be done by constructing an objectives tree, which allows specific SME objectives to be linked up to more global socio-economic objectives envisaged within the Structural Fund framework. An example of such an objectives tree for Structural Fund interventions in the domain of SMEs is presented in Box 14. The interme­ diate objectives of increasing competitiveness and promoting entrepreneurship and, to a lesser extent, Europeanisation and internationalisation of SME opera­ tions, are those most closely associated with Structural Fund interventions. By integrating objectives that are not necessarily uniquely those of Structural Fund interventions (e. g. simplifying start-up procedures and the legislative framework, etc. ), the objectives tree allows the coherence of objectives to be examined and the existence of synergies to be explicitly taken into conside­ ration. Evaluating impacts on the competitiveness of SMEs Box 14 ­ Example of a thematic objectives tree for SME interventions Global objectives Examples of specific objectives Maximise the contribution of SMEs to social and economic cohesion Improve the business environment Européanise and internationalise the operations of SMEs Increase the competitiveness of SMEs Promote entrepreneurship within special target |­ groups Improve access to venture capital Reduce administrative burdens Increase the supply of equity capital Simplify start­up procedures Simplify the legislative framework Reduce costs In trading internationally Improve Information and co­operation strategies Reduce trade barriers to third countries • Improve access to R & TD Programmes • Adapt training programmes to the needs of SMEs Encourage entrepreneurship culture Increase the development of new SMEs Increase the number of women and young entrepreneurs Links between intermediate objectives <­­ Objectives outside the Structural Funds Π • Net impacts arising from interventions in favour of SMEs Whenever possible, attempts should be made to assess the impacts of SME programmes on a net basis. This means that deadweight effects should be estimated so that the counterfactual situation can be established (e. g. the level of turnover, the amount of investment, the number of innovations, etc. that would have occurred in a policy­off situation). Furthermore, displace­ ment effects may have to be taken into account. Substitution effects may also occur and may even, in certain instances, be sought (e. g. an increase in the proportion of women and young entrepreneurs, diversification of the local productive activities towards more dynamic sectors). Transversal evaluation of impacts on intervention priorities Example: From gross to net impacts in the case of support for new business creation A scheme, part financed by the Structural Funds, to provide seed capital to finance the setting up of new businesses, has led to 60 new micro­enterprises being launched with a combined turnover of 1,36 million euro after the first year. Surveys of the addressees of the scheme found that 15 new enterprises with a turnover of 0,38 million euro would have been launched in the absence of the scheme (deadweight). It was also found that two addressees had been running undeclared businesses with a combined annual turnover of 32 000 euro. However, because this was considered as a positive effect of the intervention (i. e. the substitution of registered actives for unde­ clared activities) it was not subtracted from the gross effects. The net effect of the intervention was thus estimated at 0,98 million euro. In the case of support for investments (e. g. in physical infrastructure, capital goods, research and development, etc. ) the leverage of Structural Fund expenditure is often explicitly sought. The leverage effect, defined as the amount of private sector finance invested in an assisted project that can be directly attributed to Structural Fund co­financing of the project, should be estimated when relevant. ■ Examples of i m p a c ts by d o m a in of intervention SMEs are directly affected by interventions aimed at supporting productive investment. Furthermore, interventions targeting the provision of basic infra­ structure and the development of human resources have direct and indirect effects on SMEs. Evaluating impacts on the competitiveness of SMEs • Basic infrastructure Infrastructure investments in favour of SMEs are directly targeted at the needs of businesses, for example, interventions supporting the construction of facilities specially designed to nurture small high-tech enterprises or new businesses. This type of infrastructure is often an integral element of broader interventions in favour of SMEs. Other infrastructure interventions have an indirect impact on industry, usually by reducing environmental and accessi­ bility constraints (transport, energy supplies, information). Example: A road improvement project in Ireland Within the framework of Structural Fund interventions in the area of basic infrastruc­ ture, one of the prime objectives is to increase mobility within regions. The Athlone By- Pass project involved the construction of a 10 km dual carriageway by-pass and a river bridge. This project did not include specific measures to support industry or SMEs, but by allowing road freight to avoid the town centre, constraints on local industrial development and access to ports, and therefore to export markets, was improved. In addition to these indirect effects, during the period of construction local SMEs were directly involved in subcontracting to the main contractor of the project. Outputs > Effects > Creation of temporary employment Jobs created or maintained Construction of a new dual carriageway Reduction in delivery times to major markets Improved road accessibility for local SMEs Priority Transversal evaluation of impacts on intervention priorities • Productive environment Interventions in the area of the productive environment are those with the most direct effect on SMEs. These may include interventions in the domains of rural development and agriculture, tourism, fisheries, etc. which, whilst not specifically aimed at SMEs, are intended to have an impact on entrepre­ neurship and small businesses. Other interventions explicitly target SMEs (e. g. information services, provision of tailored facilities and on-site advice services, support for venture capital schemes, etc. ). Example: T he Ouest D o u ze Initiative (France) The Ouest Douze Initiative, located in C a p d e n a c, A u b in et Decazeville, in southern France, is c o - f i n a n c ed by the ERDF, the French g o v e r n m e n t, local b o d i es a nd the private sector. It ¡s a non-profit organisation offering technical advice a nd financial s u p p o rt (interest-free loans) to potential local entrepreneurs. It also runs a s c h e me w h e re entrepreneurs c an be t w i n n ed w i th e s t a b l i s h ed c o m p a n i es a nd w h e re past addressees return to pass on their experience to their successors on the s c h e m e. Outputs J^> Effects Support and advice sessions with potential entrepreneurs Arrangements for soft loans for the launching new enterprises New businesses formed ΐ> Increased entrepreneurial activity in the region Strengthening of SME base in the region Priority Q Evaluating impacts on the competitiveness of SMEs • Human resources Interventions concerning education, training and research in this domain may either be directed at industry as a whole or explicitly oriented towards supporting SMEs (e. g. training for individuals wishing to become self­ employed, assistance for research and development projects, brokering of research partnerships with other firms or institutes, etc. ) Example: Furniture Technology and Design Project (Finland) This project, based in Päijät­Häme in the south of Finland, aims to help increase the competitiveness of businesses in the furniture industry. To do this it supports co­ operation and networking between manufacturers, suppliers and training organisa­ tions. Vocational training schemes are tailored to the needs of individual companies and advice is given on training strategies and on anticipating future training require­ ments. The project has succeeded in increasing the level of technology and design in participating firms, leading to increased competitiveness both at home and abroad. Outputs > Employees attending courses and improving their skills Effects \ t > Participating SMEs increasing export sales Training courses designed specifically for SMEs SMEs participating in the project and improving the quality of their workforce Participating SMEs expanding into new overseas markets Priority I I ■ Defining and selecting indicators A more detailed treatment of indicators is provided in Volume 2 of the MEANS Collection devoted to indicators for monitoring and evaluation. The above Volume includes an information sheet on indicators for the priority of competitiveness of SMEs and enterprises. Interventions may be directly targeted at SMEs, others will affect SMEs indi­ rectly. The definition and selection of indicators will, to some extent, depend Transversal evaluation of impacts on intervention priorities on the type of intervention being studied. For interventions that do not focus uniquely on SMEs, but on the productive sector as a whole, a system of indi­ cators should systematically incorporate the percentage of resources being allocated to SMEs, % of projects concerning SMEs, etc. This approach cor­ responds with that of mainstreaming used to systematically highlight the extent to which the promotion of equality between men and women is being taken into account in programmes (see Box 2). Demand for many SME support services depends to a large extent on the economic context. Financial consumption will consequently tend to fluctuate over time, depending on the position of national economies in the business cycle. It is also interesting to examine the percentage of support being taken up by projects in the more dynamic or high-tech sectors. Output indicators concern the quantity of physical infrastructure or support services financed. In the case of business support services, which are of an intangible nature (e. g. information management, training or consultancy), output indicators focus on the scale of activity actually taking place and of services financed by the programme (e. g. hours of advice received by addres­ sees). They also focus on the number of addressees or projects assisted. Result indicators inform on how the supply structure of a region has been modified by the intervention by, for example, looking at the number of new business start-ups. Other indicators may be more sophisticated; they may, for example, look at the leverage effect that funding generates. Impact indicators of interventions in this domain reflect the heterogeneous nature of impacts on SMEs. They generally focus on the increased in value added arising from an intervention, knock-on effects on suppliers and sub­ sequent employment impacts that eventually arise. In the case of new firms, indicators focus on the survival rate of new start-ups. It is, however, important to be realistic about the impacts that can be observed during the timescale of an intervention. Whilst some direct impacts on employment may manifest themselves in a relatively short period, indirect employment impacts, knock-on effects, and impacts on product and process innovation usually occur only over the medium to long-term. The estimation of such impacts usually has to be based on the expectations of addressees. By way of an example, a system of indicators that could be used to track the production of impacts arising from a project providing informational support to promote exports by SMEs is presented in Boxi 5. Evaluating impacts on the competitiveness of SMEs Box 15 ­ Example of a system of indicators for an intervention providing informational support for SMEs to promote exports Programme Indicators (Related to the Intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Number of addresses Number of assisted SMEs Number of SMEs in the eligible area Result Indicators Satisfaction rate Geographical diversification Impact indicators Exports % of addressees who are satisfied or very satisfied with the support services provided Number of SMEs becoming new exporters Number of SMEs exporting to new markets % of exports sales in the turn over of assisted SMEs after 18 months Regional exports of SMEs related to GDP Value added Increase In value added generated after 18 months Average value added by employee in the region Direct employment Number of net jobs created / maintained (FTEs of which held by women) In firms In relation to export sales after 18 months Source: Volume 2 of the MEANS Collection ■ Obtaining useful information Total number of unemployed in the assisted area The extent to which an intended approach to the evaluation of SME interven­ tions can be adopted will clearly depend on the quality and availability of data and the extent to which statistical data on regional economies can be disag­ gregated to the level of assisted regions. It needs to be emphasised that some impacts are not possible to quantify, and in this case a purely qualitative approach has to be adopted. The evaluation of Structural Fund interventions in the domain of SMEs will, to a large extent, be strongly dependent upon primary data, complemented by established statistical sources. One particular problem which arises when using the established definition based on employment levels of small (0 to 50 employees) and medium enterprises (51 to 250 employees) is that a medium­sized enterprise consis­ ting of 250 employees has very little in common with one of 50 employees. Similarly a small enterprise of 50 employees is usually very different to a firm of 10 or fewer employees. When primary data is to be collected, the hetero­ Transversal evaluation of impacts on intervention priorities geneity within the overall category of SMEs can be dealt with through careful sampling and choice of case studies. However, established data sources often do not collect data on enterprises of less than 20 employees - a popu­ lation which for certain types of intervention (i. e. financial support aimed at assisting small firms to break through the 20 employee barrier or the creation of micro-enterprises) may be of particular interest. Another problem with secondary data is that while the "birth" of a new enterprise is immediately registered when the owner goes through the relevant administrative proce­ dures, the "death" of a firm may not be recorded until a long time after it has terminated its activity. This may lead to an over-estimation of overall SME activity and to the inclusion of non-existent firms in analyses. Many context indicators, such as business start-up, may be available from European and national administrative records, statistics and surveys (e. g. those collected for taxation purposes). Eurostat has developed a special regional database, called Regio, of regional industrial statistics with annual Europe-wide comparable data dating back to 1994. In conjunction with DG XXIII, Eurostat publishes the report "Enterprises in Europe" at regular intervals. This report contains detailed information by sector of economic activity and by payroll size classes on all the enterprises in the European Union (on the basis of their number, workforce and turnover), as well as thematic analyses (the demography of enterprises, innovation in SMEs, labour costs and productivity in SMEs, etc. ). An annual publication of the European Observatory for SMEs gives an overview of the performance and perspectives of SMEs in all Member States of the European Economic Area and Switzerland since 1993, and provides structured data on SMEs in both quantitative and qualitative terms as well as a number of detailed thematic studies (tourism, environment, etc. ). The BACH database, based on detailed balance sheet data and covering 23 manufacturing sectors in 1987-1995 in 12 EU countries, contains data on SMEs. Data is broken down by different size groups based on a harmonised definition. The information is provided by central banks and statistical agencies. The database is particularly well suited to financial analysis. Evaluating impacts on the competitiveness of SMEs ■ Appropriate tools for analysing impacts on SMEs While a detailed treatment of evaluation tools is provided in Volume 3 of the MEANS Collection, a number of tools particularly suited to evaluating impacts on SMEs are presented below. An assessment of the impact of interventions on SMEs can be undertaken using the "step­by­step" approach presented in Part IV of this Volume. Face­to­face interviews can be used to examine issues concerning the strategic aspects of Structural Fund interventions. They might be carried out with key actors in national authorities and managers who have overall res­ ponsibility for SME support programmes in the Member States. A disadvan­ tage of this technique is that it can be very time consuming when a large number of interviews have to be organised. When a large number of people sharing a collective experience (e. g. in the implementation of interventions) have to be interviewed, the use of focus groups may be less time consuming and more useful. However, it should be kept in mind that whilst the use of this tool may allow useful qualitative information about implementing inter­ ventions to be gathered, it is not suited to all situations (e. g. where a detailed description of processes is required, individual interviews are more likely to produce more useful information). Questionnaire surveys allow a basic assessment of the impacts of measures and give insight into the contribution of different schemes to SME competiti­ veness and growth as well as the implementation of programmes (i. e. the accessibility and delivery of EU­funded schemes). When a questionnaire is well designed, it will allow reasonably definite judgements to be made on the extent to which improved performance can be linked to assistance from EU­ funded schemes. Whilst it is less expensive and time consuming to rely on monitoring data for constituting samples, this presents a problem of sampling bias. Monitoring data can effectively include non­addressee firms (i. e. firms which have made contact with the operators but which have not subsequently become addres­ sees). However, using information on these enterprises to construct the coun­ terfactual situation ignores the behaviour of non­addressee firms that, for a number of reasons have not approached the authorities with regard to the programme (e. g. they may be unaware of a programme's existence, they may perceive that the cost of applying for assistance outweighs the benefits, etc. ). Factor analysis is particularly useful for analysing the characteristics of large homogenous populations. Because SMEs are highly differentiated, this Transversal evaluation of impacts on intervention priorities technique can be used to generate clusters of firms according to their key cha­ racteristics. Impacts can then be analysed in relation to these characteristics. A factor analysis is usually fed with data from questionnaire surveys. This technique was successfully used in Spain when evaluating the employment impacts of job subsidies in the Catalonian co-operative sector (see Box 16). Box 16 - Example of a factor analysis of the Catalan co-operative sector The use of factor analysis can be illustrated with respect to an ex post evaluation project, oriented towards the employment effects (creation and maintenance of jobs) of subsidies in the sector of Catalan co-opera­ tives. Evaluation in this case was complicated by the diversity of co-ope­ ratives and the consequent heterogeneity of the population. The problem was twofold: first, the relevance of measures taken by the region had to be determined in view of this situation; secondly, the importance of impacts on those co-operatives which received aid had to be assessed. Given the variety, as well as the lack of knowledge of the population and of impacts, factor analysis was chosen for its ability to reveal the main characteristics of the population under consideration (about 3,000 co­ operatives). The first step in the evaluation consisted of sending a questionnaire to all the co-operatives. This was completed by interviews. The data thus gathered were first analysed using traditional statistical methods (search for the main characteristics of the population, drawing up of cross-refe­ renced tables). The second step in the data analysis consisted of a factor analysis with a view to isolating variables likely to explain variations in employment in the co-operatives: including the size of the entity, age, sector of activity, technological level, etc. Source: Volume 3 of the MEANS Collection Surveys can be prepared or complemented by a number of case studies which are able to uncover impact causation of SME measures and to test, justify or qualify results obtained by surveys or other methods. They might focus on particular types of Structural Fund interventions (e. g. financial engi­ neering, human resources, business support services, innovation and tech­ nology, etc. ). It is important, however, to carefully select case studies to allow useful in-depth analysis and comparison between individual projects. Although the selection is strongly dependant upon the willingness of sponsors and partner organisations responsible for the action to participate in the research, and on the availability of quality information (including moni- Evaluating impacts on employment toring data), reasonable coverage of the various types of actions is desirable. The selection of case studies can be guided by taking into account the SME initiatives that involve major expenditure, and the impor­ tance of different SME actions from the point of view of regional develop­ ment strategies. For an example of the use of case studies see Box 17. Box 17 - Example of the use of the case study technique in the evaluation of the NordTek programme, Denmark This programme, intended for small and medium-sized firms, consisted of about 100 projects offering services to firms in the areas of training, consul­ tancy, access to modern computer equipment, etc. It was therefore a programme that could be qualified as "soft", as opposed to "hard" pro­ grammes aimed at improving infrastructures. The results of such pro­ grammes are difficult to evaluate precisely because they are not tangible. Six large projects were selected as cases to be studied. In terms of the selection criteria, the projects had to be representative of the diversity existing in intervention strategies and locality. The selected projects also had to have benefited from significant funding in the framework of the NordTek Programme and be oriented exclusively towards firms (rather than research institutions or administrations). The six projects evaluated repre­ sented such a large portion of the total Programme that the analysis of the impacts of these interventions could be used to cover all the important phenomena. It was recognised without ambiguity that the selected projects represented the "best cases" of the programme. The data were collected at two operational levels: at the level of the project as such, and at that of the firms. As far as the project was concerned, the evaluators studied each of the cases in the field four or five times over a period of two years. Each time the managers were questioned and each project had to keep a record, in a "register" designed especially for the purpose, of the services proposed and the firms concerned. The analysis of the documents thus obtained was handed to the project leaders and discussed during visits. As for the firms, the data were collected during interviews with a sample of 20 SMEs which had used the services offered. Finally, 40 SMEs who had not used the proposed services were questioned in the final stage of the evaluation. Source: Volume 3 of the MEANS Collection Transversal evaluation of impacts on intervention priorities ■ S u m m a ry The important role of small and medium­sized enterprises (SMEs) in suppor­ ting growth and job creation in the European Union has been brought into sharper focus over the past few years. The flexibility and innovative capacity of SMEs enable them to respond quickly to structural changes and adapt to changing demand. However, there are a number of factors that hamper the competitiveness of European SMEs and basic market and policy imperfec­ tions that inhibit growth and job creation. The evaluation of Structural Fund interventions in favour of SMEs therefore has a major role to play in assessing the extent to which stated policy objectives have contributed to overcoming these problems. Interventions may focus directly on SMEs or on SMEs as a sub­population of industry. SMEs are also implicitly targeted by interventions in other domains, especially those where diversification of productive structures is an objective (i. e. rural development, fisheries and tourism). As far as suitable empirical techniques for evaluating Structural Fund inter­ ventions in favour of SMEs are concerned, experience of their application to this domain is limited, but it is likely to accumulate rapidly as evaluations of this type of intervention become more commonplace. ■ For m o re information see. EIM Small Business Research Consultancy (1997), European Observatory for SMEs: Fifth Annual Report, The Netherlands: EIM. European Commission. (1995), Activities in favour of SMEs and the craft sector, Luxembourg: Office for Official Publications of the European Community. European Commission. (1997), Report from the Commission to the European Parliament, the Council, and the Economic and Social Committee and the Committee of the Regions on the Coordination of Activities to Assist Small and Medium­sized Enterprises (SMEs) and the Craft Sector. Grant Thornton International Business Strategies Ltd. (1998), European Business Survey: Prospects and Issues for SMEs, Spring (6). OCDE. (1993), Les Petites et Moyennes Entreprises: technologie et compéti­ tivité, Paris: OCDE. OECD. (1998), Besf Practice Policies for Small and Medium­sized Enter­ prises, Paris: OECD. OECD, (unpublished), Assessment of SME Policies: An Analytical Framework, Working Document, DST/ IND/ PME (98)2. Evaluating the impact of structural interventions on employment ■ Why do employment impacts need to be evaluated? • The Community context Structural unemployment is a persistent problem with which the European Union has to contend, even during periods of strong economic growth. Those who are unemployed experience a loss which is more than financial and the wider community has to bear the burden of supporting an economi­ cally inactive section of the population, the costs of increasingly established pockets of long­term unemployment and the associated deprivation which tends to be concentrated in both social and geographic terms. The response of the European Union to this situation is based on the premise that a Union­wide effort is needed to establish a both a micro and macro­economic context that favours the creation of employment. This effort must simultaneously co­ordinate and work in partnership with initia­ tives and action taken on a national, regional and local level. The Structural Funds are used to co­finance policy interventions which aim not only to promote a better economic and social balance across the Union by reducing regional and national disparities (cohesion), but also to assist the overall development of the Union's structural fabric, including that of the labour market. They are intended to stimulate economic growth and the creation of employment. Although some of the objectives of structural policies have always been explicitly related to employment, in recent years a growing concern with producing substantial employment effects has become increasingly evident both in principle and in practice. As a major ins­ trument for influencing economic development patterns and a major vehicle for the delivery of resources to local partnerships, it is evident that the use of the Structural Funds needs to reflect the increased priority which the Union has given to employment. The types of employment impacts likely to be produced by interventions assisted by the Structural Funds are critically influenced by the particular instruments used in each case. The choice of such instruments is influenced in turn by the objectives of the intervention in question. An intervention in the area of the productive environment or basic infrastructure may aim to generate employment effects by creating new jobs or maintaining existing Transversal evaluation of impacts on intervention priorities ones. On the other hand, interventions in the area of human resources focusing on training are targeted at enabling individuals to meet employabi­ lity (i. e. the ability of an individual to meet the technical and organisational requirements of employment available at a given time and in a given place). The addressees of these interventions may be for people already in work who are adapting to changing work conditions, or the unemployed, particu­ larly special target groups such as women, the long term unemployed and young people, who need to improve their employability by enhancing existing skills or developing new ones. A defining characteristic of structural interventions is that they are concen­ trated on a limited number of global objectives, and each programme of interventions assisted by the Structural Funds is concentrated on achieving one of these objectives. Global objectives concern both specific regions and specific social groups and in either case they may be defined in relation to a specified economic sector. The relationship between the different global objectives and the priority of promoting employment varies considerably. • Accurate evaluations of the effectiveness of Structural Fund interventions The potential benefits arising from effective Structural Fund interventions in the area of employment are substantial: a reduction in the poverty and misery that unemployment causes, the creation of labour supply conditions that promote growth and competitiveness in the weaker regions, and an increase in social and economic cohesion. Ultimately these benefits could also include increased European-wide competitiveness and the underpin­ ning of the European social model. However, if Structural Fund interventions are to be shown to be effective, evaluations will need to be sufficiently sophisticated to measure employment impacts accurately. If interventions are to be effective, the ex ante and ex-post measurement of employment effects has to be as accurate as possible. On a more strategic level, accurate information allows the measurement of the overall impact of interventions on employment, permitting informed decisions to be made by the European Commission and Parliament on the continuation or modifica­ tion of Structural Fund interventions. Since Structural Fund interventions combine programmes of measures with a variety of objectives, there is a particular need to evaluate not only the overall employment effects of these programmes, but also the impacts of Evaluating impacts on employment each component measure. Moreover, in many cases it is necessary to evaluate both the general impacts on employment (e. g. how many new jobs have been created?) and impacts related to more specific objectives (e. g. how many jobs are now concentrated in a particular region or held by young people? - regardless of whether the total number of jobs has increased). To fully appreciate the effectiveness of interventions it is of the utmost impor­ tance that net impacts are estimated. Consequently, this means that the creation or maintenance of employment that would have taken place in the absence of an intervention has to be taken into account. Similarly, allo­ wances must be made for employment that is displaced elsewhere in the programme area and for individuals who are eligible for assistance who are substituted for non-addressees. Finally, the wider repercussions of an inter­ vention on suppliers of recipient firms and the increased purchasing power of assisted individuals should be estimated when this is feasible. Continuous interaction between labour markets and other types of markets (i. e. goods, services or financial markets) or markets outside the defined geographical area also has to be taken into account. For example, an increase in exports due to favourable economic conditions overseas would have a positive effect on employment creation. Another example could be a period of relatively low interest rates that may have positive effects on investment and exports with consequent benefits for employment. These confounding factors should be taken into account if the effectiveness of an intervention is to be accurately measured. Furthermore, numerous interventions often occur at a national level. As a result, synergies may often be generated between Structural Fund interven­ tions and national initiatives. It is therefore useful to be able to identify the existence of such synergies. Certain interventions are specifically focused on social groups and, in parti­ cular, the employability of individuals in employment or unemployed. Employability is closely linked to the evolution of the skills of individuals to reflect the changing needs of the economy. The problem of maintaining employability is particularly acute in former industrial regions where profound changes in the organisation of production and corporate strategies have occurred as a result of structural change, leading to the obsolescence of workers' skills. Box 18 presents several forms of skill obsolescence and cor­ responding training strategies for increasing the employability of individuals. Transversal evaluation of impacts on intervention priorities Box 18 - Different types of skill obsolescence Form of obsolescence Related to the individual Related to the conditions of the post Related to changes in the post Related to the firm Related to to the market Description Age, Illness, injury Insufficient use of skills Demands related to the type of job change Appearance of new compe­ titors undermi­ nes a business Employment in a given sector declines Response in terms of training Revision of skills already acquired or total re-quallfication Revision of skills already acquired Training for new requirements of the job Development of existing skills or total re-qualification Training for similar jobs in In a new sector or total re-quallfication In order to be clear about the type of employment effects under discussion, reference will be made throughout this chapter either to job creation or main­ tenance effects produced by interventions in non-human resource domains or to employability effects produced by human resource type interventions (mainly concerning training). The essential difference between these two types of effect is that in the first instance employment effects concern the creation or maintenance of a post which may be occupied over time by a number of individuals. In the second, employment effects concern maintai­ ning or improving the employability of individuals so that they can either remain in a post at a certain level of qualification (i. e. in the case of training of workers in employment), or have a better chance of matching the skills requirements of potential employers and therefore of finding a job (i. e. in the case of training for the unemployed). The characteristics of these different types of employment effect are summarised in Box 19. Box 19 - Breakdown of different types of employment effects Employment effects Job creation / maintenance effects (Related to a post) Employment created or maintained by the process of implementation of SF interventions or as a consequence of interventions where job creation or maintenance is an explicit or implicit objective Employability e f f e c ts (Related to an individual) Employment maintained (existing employees) or found (unemployed persons) as a consequence of SF interventions targeted specifically at maintaining or improving the employability of individuals through the development of skills and qualifications Evaluating impacts on employment ■ Defining what needs to be measured How employment impacts are produced Many interventions are not explicitly targeted at the creation of employment effects; they are therefore unlikely to have a substantial direct effect on employment. Other interventions are targeted at the development of human resources, including training and research which produce only limited direct employment effects in the short term. Consequently, the most significant employment effects resulting from the use of the Structural Funds are often indirect and tend to arise in the medium to longer term. They can therefore be relatively difficult to isolate and measure. This situation is further compli­ cated by the fact that certain interventions may well create negative employ­ ment effects initially, before any positive longer term impacts occur. This is particularly the case with interventions targeted at increasing productivity and competitiveness. The ways in which structural interventions may influence employment patterns are listed in Box 20. While it may be argued that the key analytical distinction to be made in relation to employment effects is whether they result from changes on the supply­side or the demand­side of the economy, the types of effects listed serve as a more practical reference to ensure that no effects are overlooked. The employment effects presented in Box 20 are situated at different points along the chain of causality, it is important to understand the complexity of the causal mechanisms that link interventions with their eventual employ­ ment impacts. Transversal evaluation of impacts on intervention priorities Box 20 - Production of employment effects through structural interventions • As a direct consequence of the implementation of an intervention (e. g. construction, administration and training jobs), • As a direct consequence of assistance provided i. e. through job creation or maintenance in companies which have been established or which have grown, • As a consequence of training measures targeted at the employability of existing workers and the unemployed, • As a consequence of interventions, which may or may not have an explicit and immediate employment objective, either through furnishing an intervention with goods and services (e. g. providing building materials, office supplies, etc. ) or as a result of improved efficiency in the local economy (e. g. hauliers employed to ship goods, the export of which have been made competitive due to decreased transport costs resulting from the use of new infrastructure), • In response to increased demand generated by any increase in income resulting from the intervention ("income multiplier effects"- e. g. increase in local retail employment or local house construction to meet the increased demand from the newly employed). These mechanisms are presented in Box 21 and detailed in the resume that follows it. Box 21 - Structural Fund interventions and employment effect mechanisms Low complexity of causal mechanisms High complexity of causal mechanisms (a) Construction and implementation (b) Employment subsidies (c) Training (d) Support for productive activities H Infrastructure and increased competitiveness (a) Construction and implementation leading to job creation Almost all structural interventions will have employment as a side effect in terms of construction jobs (e. g. workers employed for the construction of a motorway) or implementation jobs (e. g. teachers recruited for training pro­ grammes). These effects may range from the almost insignificant (i. e. certain financial support measures), to the major part of effects (i. e. construction of physical infrastructure). In this case the link between the interventions and the employment effect is also fairly direct. (b) Employment subsidies leading to job creation Employment subsidies for the long-term or young unemployed are a typical Evaluating impacts on employment example of policies producing employment effects as an immediate result. It is quite straightforward to connect the recruitment of a new worker to the subsidy given to the employer. (c) Employment effects as a consequence of the improvement of skills Training policies are an example of measures that can have a complex link with employment effects. The rationale for training policies is that there is a mismatch between the skills of the available labour force and those needed by sectors with good growth prospects. Training is then supposed to ensure the employability of the unemployed or of workers whose skills are likely to become outmoded. In the case of the training for existing workers, employ­ ment effects are relatively direct. Less directly, this may lead to an improve­ ment in the quality of the workforce causing both productivity and the supply of trained workers to increase, moderating pressure on wages and inducing firms to eventually increase their level of activity and employment. (d) Induced or facilitated employment effects through support for productive activities This category encompasses all jobs created by companies or other economic actors in response to the policy, through the creation or growth of productive activities supported directly (with investment grants or subsidies) or indirectly (with services, provision of sites etc. ). It is therefore the most important channel through which employment effects occur. However, along with other types of interventions that create employment effects through complex causal mechanisms, impacts on employment are difficult to attribute and measure. This is partly due to the fact that as the impact chain from intervention to employment effects becomes more complex, numerous factors, other than the support scheme, will be considered by an economic actor before taking a decision on employment. A significant example to consider here is the employment impact of invest­ ment subsidies to firms. The resulting increase in the level of investment should lead to an increase in the economic activities of the firm and to a consequent increase in employment. However, if demand is static or declining, there might well be a loss of employment. This will happen because any measure directed to the better use of capital equipment will tend to increase the productivity of fixed capital, making labour less competitive. In conditions of static demand this will lead to the substitution of labour by capital. Transversal evaluation of impacts on intervention priorities Support for productive activities also has indirect effects (income multiplier and supplier effects) with consequent employment effects. Further steps have to be taken along the impact chain and it is thus more complex to disentangle such effects. (e) Employment effects as a consequence of increased competitiveness arising from new infrastructure It is also possible that employment effects might arise from the increased competitiveness of the assisted economy owing to material (e. g. roads, tele­ communications etc. ) and immaterial (e. g. business services, research and development etc. ) infrastructure. These long-term effects are, however, produced by a complex and indirect mechanism, situated close to the end of the impact chain, and are difficult to identify and connect to a specific interven­ tion. Not only do the Structural Funds produce employment effects in a variety of ways, but the types of employment effect produced also vary. In addition to the distinction between job creation or maintenance effects and employabi­ lity effects, two other important distinctions can be made. When carrying out an evaluation it is necessary to distinguish between sus­ tainable (or durable) employment (i. e. jobs resulting from an intervention which will continue in the absence of public support) and employment which are likely to be of a transitory nature (i. e. jobs which cease to exist when the funding stops). Moreover, while employment in the implementation of an intervention is usually temporary, experience has shown that only a propor­ tion of so-called permanent jobs can be expected to last, even in the medium-term (e. g. a significant proportion of jobs created in newly establi­ shed enterprises cannot be expected to survive for long). When measuring employment effects, it is important to specify the nature of the jobs created or maintained and not to give a false impression of the long-term security of any jobs which are a consequence of the intervention. Some jobs are only created for a short duration (temporary employment), and are often related to the implementation of the intervention. If aggregated figures are to be produced then, rather than simply counting the number of jobs that are created by an intervention, it is more accurate to use expected job-years as a unit of measurement. Time discounting of expected job years can be used to estimate their present value. Evaluating impacts on employment Furthermore, the employment effects of an intervention may include both part-time and full-time jobs. Once again this distinction needs to be made. Consequently, two different types of figure are useful: first, full-time equiva­ lents (FTEs) which is the aggregation of part-time employment so that it is expressed as the equivalent number of full-time positions (e. g. one half-time and two quarter-time jobs together being considered as the equivalent of one full-time job) and, secondly, the simple number of jobs generated (e. g. one half-time and two quarter-time jobs together being considered as the equivalent of three jobs) which better reflects the objective of combating unemployment. • Clarifying employment objectives When evaluating the employment effects of interventions, reference needs to be made to the objectives of the intervention in question and how they relate to employment. In practice, neither the objectives of each component of a programme nor the connections between different levels of a programme are always articulated as clearly and explicitly as they might be. The need for clarifying the multiple levels of objectives in a programme supported by the Structural Funds is all the more necessary when the uncertain implications for employment of individual types of intervention are considered. For example, improvements in transport infrastructure serving remote regions might lower the costs for exporters from such regions and allow easier access to support services, but they might equally expose the regions concerned to even stronger competition and facilitate the seasonal or permanent emigration of the labour force. Similarly, assisted training schemes should generally increase workforce qualification levels, producti­ vity and the attractiveness of a particular location for inward investment, but in some circumstances they might lead to increased emigration of the newly qualified in search of better career prospects. If the nature of the employment effects expected from an intervention are to be assessed and appropriate techniques for their measurement applied, it is important for the Commission to work closely with those partners involved on the national, regional and local level. Together they should make the specific objectives relating to employment effects at all levels of each programme as explicit and clear as possible. This task is of particular impor­ tance during the ex ante phase of programming. Transversal evaluation of impacts on intervention priorities However, it is commonly found, especially in the cases of programmes which run over a relatively long period, that absorption problems or changing economic situations can lead to a need for refocusing in order to better achieve their desired impact on employment, or for that matter on any other priority, and to refine objectives originally specified in individual programmes. Objectives related to employment must therefore not only be clearly specified at the outset of programmes, they must also be open to modification in light of new detailed political priorities or changing economic conditions. The construction of an objectives tree, preferably at the ex ante phase, can be used to clarify the employment objectives of a programme. By way of an example, a tree of employment objectives has been constructed to illustrate the use of this tool in the context of interventions to promote employment (see Box 22). Box 22 - Example of a thematic ctives tree for the promotion of employmc abje int Global objectives Specific objectives Reduce disparities in unemployment between the regions Increased employment potential of SMEs Increased employment opportunities • Promote new business creation • Promote investment by SMEs • Promote innovation by SMEs • Promote collaboration between SMEs and research institutes • Improve consultancy services for SMEs • Diversify the range of products and services of regional firms • Increase export opportunities for regional firms • Attract Inward investing enterprises Increased employability of workers already In employment • Adapt existing employees skills and qualifications • Develop professional training activities Increased integration of special groups into the labour market • Retrain the long-term unemployed • Improve the qualifications of young people • Improve training opportunities for women Evaluating impacts on employment In this example, which is in fact a composite of different types of interven­ tions not normally found within the context of a single programme, interven­ tions are targeted at a range of specific objectives which themselves are subordinate to four intermediate objectives: increasing employment oppor­ tunities, increasing the employment potential of SMEs (interventions targeted at the productive environment), integrating special groups into the labour market (interventions targeted at training for the unemployed), and increasing the employability of existing workers. The global objective is to reduce regional disparities in unemployment. • Estimating net employment impacts of an intervention It is important that only those impacts that genuinely arise from an interven­ tion are attributed to it. To do this, the net employment impact of an interven­ tion should be estimated. A first step in estimating net employment impacts is to construct the coun­ terfactual situation in order to estimate the proportion of gross employment impacts which would have occurred in the absence of an intervention (i. e. deadweight). The gross employment impact of an intervention is defined as the total number of jobs created or maintained or the number of people passing through training courses, counselling sessions, etc. who remain in employ­ ment or find a job. In practice, data on gross employment impacts is collected by monitoring systems and tends to be used as the basis for this figure. When interventions focus on training either existing workers or the unem­ ployed, the deadweight effect is defined as the number of people passing through a scheme who would have remained in employment or found a job without receiving the training. The counterfactual situation in this case is usually estimated by establishing a control group. The profile of addressee employees is important in establishing the extent of this type of deadweight effect. For example, the training of management or other highly skilled staff is more likely to occur without the support of interventions. Similarly, the size of the firm may be a determining factor, since larger firms tend to undertake more training activities than SMEs. Deadweight is subtracted from the gross impacts of an intervention. Displacement can occur in the case of area-based interventions, when the creation or maintenance of jobs takes place at the expense of jobs elsewhere Transversal evaluation of impacts on intervention priorities in the programme area. An example of displacement is of companies assisted by an intervention expanding their business and their workforce by taking business from other companies within the programme area. Displace­ ment should be subtracted from gross impacts when the effect is unintended. When interventions are targeted at particular groups of addressees (e. g. long term unemployed, working mothers, young people, etc. ), the substitution of addressees for non-addressees may in fact explicitly be sought, for example, with better trained addressees finding employment at the expense of non-addressees on the labour market. Similarly, interventions which expli­ citly aim to create or maintain employment by supporting the development of a sector at the expense of another can be considered as substitution. When substitution is the explicit objective of interventions, it should not be subtracted from gross impacts. If, however, the intervention was not expli­ citly aimed at creating this effect, substitution should be subtracted from gross impacts. Example: A training scheme for business creation: from gross to net effects A training scheme for business creation might lead to the creation of some 400 busi­ nesses of which only 250 businesses were still active two years later, with 50 of these employing the owner half time only. Assuming that surviving businesses created 350 additional jobs (since the aim of the intervention was to create sustainable jobs, it was considered that two years was an adequate time scale to judge sustainability), the gross number of jobs at the end of this period would amount to 300 (employees) plus 250 (business owners), a total of 550 gross jobs. However, because 100 jobs are half- time, the number of gross jobs created amounts to 525 FTEs (200 + 50 / 2 + 250 + 100/2 = 525). If, however, deadweight is estimated to be 20%, and the displacement of businesses because of the support given to recipient businesses is estimated at 30%, the net effect of the intervention will be 262. 5 net jobs in terms of FTEs (525 - [525x0. 3] - [525x0. 2] = 262. 5) The differences between these concepts as they relate to job creation or mainte­ nance effects or employability effects are summarised in Box 23. In addition to these different effects employment may be generated through income multiplier effects (i. e. employment which is created / maintained indi­ rectly by the fact that the intervention results in additional income for local consumers) or supplier effects (i. e. employment which is created or maintai­ ned as a consequence of backward linkages between addressees and their suppliers) which tend to be separately identified. However, these effects are virtually impossible to quantify for individual regional interventions. Evaluating impacts on employment Box 23 - Comparison of deadweight, displacement and substitution effects Definition for job creation / maintenance Jobs which would have been created or maintained in the absence of the Intervention Jobs created or maintained by a company in the area targeted at the expense of other companies In the programme area Deadweight Displacement (area-based interventions) Substitution (social group-based Interventions) Definition for employability Retrained individuals who would have remained in employment or found a job In the absence of the intervention Companies employing or recruiting retrained individuals which take business and reduce employment In other companies in the programme area which have not retrained their employees or recruited retrained people Retrained individuals who remain In existing posts instead being replaced by previously better qualified unemployed persons Retrained unemployed individuals belonging to specific target groups who are substituted for non-addressees The impact of an intervention, either positive or negative will not always be directly translated into an effect on unemployment of a similar magnitude. This is because any changes in the number of people actively seeking employment need to be taken into account. One such effect is the impact on activity rates that is induced by new job opportunities, the so-called attraction effect (or added worker effect). For example, new part-time jobs may attract women or retired people who were not registered as unemployed before the intervention occurred. Another effect on the supply of labour is that of migration: new job opportuni­ ties may have an impact on both immigration (by attracting some individuals to move into the target area to obtain work) and emigration (by reducing the number who would otherwise leave the target area in search of work). In this context, some effects on travel-to-work patterns may be considered as migration effects and these may be even more significant than more permanent migration flows since greater numbers of people outside the target area may be encouraged by new job opportunities to simply commute into the area. The smaller the target area, the more likely it is that new jobs will go to individuals who were not part of the local supply of labour before the intervention. Transversal evaluation of impacts on intervention priorities ■ Examples of impacts by area of intervention In order to illustrate how interventions can lead to employment impacts, examples are presented below which correspond with the three broad areas of intervention of Structural Funds. • Basic Infrastructure The aim of interventions in this area is largely to relax constraints on economic development resulting from the absence or poor quality of basic infrastruc­ ture. Such interventions do not, therefore, have explicit employment objec­ tives. Job creation and maintenance effects are either of a temporary nature during implementation, or indirect due to increased economic activity. Example: T he upgrading of analogue telephone e x c h a n g es (Greece) The public s w i t c h ed t e l e c om network in Greece w as almost 1 0 0% analogue at the beginning of the 1990s. As a result, the network suffered from low quality and reliabi­ lity. One of the interventions designed to improve the network has been the progres­ sive replacement of existing analogue e x c h a n g es w i th new digital e x c h a n g e s. T he objective of this p r o g r a m me is to reach a level of 6 8 ­ 7 0% digitisation of e x c h a n g es by the year 2 0 0 0. As e x c h a n g es b e c o me d i g i t i s e d, f i r ms a nd h o u s e h o l ds are able to benefit g l o b a l ly f r om i m p r o v e m e n ts in t he n e t w o r k. T he m o d e r n i s ed n e t w o rk will p r o v i de f i r ms w i th t he o p p o r t u n i ty to t a ke a d v a n t a ge of t he n ew p o s s i b i l i t i es for business offered by t e l e m a t i c s. While m o st e m p l o y m e nt effects will be t e m p o r a r y, occurring during i m p l e m e n t a t i o n, over the longer term it is possible that permanent e m p l o y m e nt opportunities will arise as firms take advantage of the market possibilities that the improved infrastructure offers. Outputs ~^> Effects Telephone exchanges re­equipped Gross temporary jobs created or maintained Gross jobs created or maintained Reduction In network faults Firms potentially served by modernised exchanges Firms using telematics to expand into new markets Increased production, sales, etc. Priority Evaluating impacts on employment • Productive environment For some regions the aim of Structural Fund interventions is to assist the reconversion or diversification of the economy in order to combat structural unemployment. In rural regions, diversification of productive structures to reduce dependence on agriculture and to encourage tourism, a highly labour-intensive activity, is an essential element in a strategy of economic development. Example: Developing tourism in the Republic of Ireland In Ireland, an Operational Programme in Tourism was implemented from 1990 with the generation of employment effects (the creation of an additional 25 000 jobs) as its global objective. One of the specific objectives of interventions was to diversify the range of tourism products. One of the product themes chosen as a suitable target of intervention was the development of conference travel to identified tourism centres. During the 1989 to 1993 period twelve new facilities for conferences were co- financed. The construction of conference facilities produced temporary employment effects and an eventual impact on the growth and seasonality of the activities of local businesses such as hotels, restaurants, conference services, etc. Outputs > Effects Gross temporary jobs created or maintained Construction of conference centres Organisation of conferences and events Increase in non-seasonal and bus mess tourism 1 New activity generated for the region's businesses t Priority J\ , > 1/ Gross jobs created or maintained k , Transversal evaluation of impacts on intervention priorities • Human resources Interventions in the area of human resources focus on maintaining or improving the employability of individuals through enhanced skills and abilities, and on supporting Research and Technological Development. Example: Support for vocational training in t he North East of England The North East of England, o n ce a major industrial region of the UK, has lost over 200 000 j o bs in traditional sectors (coal mining, shipbuilding, steel making and heavy e n g i­ neering) since 1970. At the s a me t i m e, high t e c h n o l o gy activities in the region have d e v e l o p ed in such s e c t o rs as pharmaceuticals, biotechnology, information t e c h n o­ logy and electronics. One of the factors contributing t o w a r ds the emergence of new specialisations is the strong links b e t w e en industry and the region's universities and o t h er higher e d u c a t i on i n s t i t u t i o n s. S o me of t he m e a s u r es in f a v o ur of industrial reconversion of the North East (CSF 1989­91 and 1992­93) are aimed at vocational training. These measures include the provision of facilities for training in industrial sectors with a high g r o w th potential and especially in technologies having an indige­ nous base in the region. Outputs > Effects C> Gross temporary jobs created or maintained Gross jobs created or maintained / Training projects prepared in partnership with local industry Unemployed trainees J being offered employment during the course Trainees successfully completing the course ^ Unemployed trainees finding employment λ * Recipients promoted. Employees remaining in employment Priority ■ Defining a nd selecting indicators Volume 2 of the MEANS Collection is devoted to the subject of indicators for monitoring and evaluating Structural Fund interventions. This Volume includes an information sheet on indicators for the priority of employment. Whilst evaluations are not limited to an analysis of indicators alone, they are greatly facilitated by the existence of relevant indicators. The subject of indi­ Evaluating impacts on employment cators and their quality needs to be considered at two different levels in relation to employment effects: • Since the promotion of employment is one of the general objectives of the Structural Funds, the ongoing development of a system of indicators should result in a strengthening of information on the relative employment effects of the different components of a programme. • The issue of indicators also arises in relation to more detailed analyses of effects. Any evaluation which focuses on the subject of employment effects requires ad-hoc employment indicators to be defined and selected in relation to the specific evaluation questions being asked. In order to evaluate the employment effects of the Structural Funds, it is conse­ quently useful, first, to ensure that some basic and reliable indicators of employ­ ment are included in a programme's system of indicators and secondly, in the context of any particular evaluation exercise, to define those central indicators which respond to the specific evaluation questions being posed. When measuring employment effects, account should be taken of the temporary jobs needed to implement the interventions (e. g. jobs related to the building of infrastructure, advisory services for businesses or training). Whilst output indicators do not focus directly on employment, they are usually essential for an analysis of the production of employment effects. This is particularly the case when interventions result in the provision of services such as training, information or advice which, due to their intangible nature, can be difficult to measure. Effort should be concentrated on the establishment of those indicators that provide information on the objectives which are most significant as far as employment is concerned. These indicators refer for the most part to gross direct effects. The objectives of employment-related structural interventions are diverse and the definition of systems of indicators must incorporate this diversity. There is no one universal indicator of employment effects which is relevant regardless of the type of evaluation being carried out. For example, if the aim of an intervention is to promote the return to work of those who are unem­ ployed in an urban area that is being redeveloped, the number employed in this urban area will not in itself be sufficient as an indicator. Since it does not enable the proportion of jobs occupied by residents of the area to be deter­ mined, the effectiveness of the intervention cannot be judged. Transversal evaluation of impacts on intervention priorities Finally, selected indicators must be precisely defined. For example, the defi­ nition should state whether a job created or maintained includes all types of job, or are part-time and full-time jobs distinguished? How is a job is qualified as sustainable? Is it a job that is expected to remain in existence for more than two years, more than five years, etc. ? How are specific social groups targeted by interventions defined? For example, the information "30% of the jobs created by the project benefited marginalised people" is of little use if the concept of marginalisation is not clearly defined. • Indicators for a more in-depth analysis of employment effects Apart from the central employment indicators in a system of indicators, it is often relevant during an evaluation to use ad hoc indicators in order to make a more in-depth analysis, particularly of impacts. Any such use of supple­ mentary indicators needs to be consistent with the questions that the eva­ luation is required to answer, the particularities of interventions and the cha­ racteristics of the region being evaluated. Supplementary indicators might therefore focus on: • the proportion of jobs created or maintained which are unskilled / skilled / highly skilled; • the remuneration of jobs created or maintained (e. g. the effect will not be the same if 10 jobs remunerated at the legal minimum rate or 10 higher level posts are created); • the personal earnings of addressees; • the profile of the addressees (even when the objectives do not refer to target groups, it may be interesting to know who is benefiting from job creation). An assessment of compliance with the principle of equal opportu­ nities is relevant at this level; • the career paths of addressees. In addition to knowing that they have been re-employed, it may be interesting to know, for example, how long it took them to get a job, how appropriate the job is in relation to their qualifications, or even what effect the intervention had on their approach to seeking work; • the type of the jobs created or maintained. Making a distinction between part-time and full-time jobs is one way of making inferences about the quality of jobs created or maintained. However, such inferences have to take into account the fact that some addressees may prefer to work part-time rather than full-time. Differences in national employment and social security legislation are also key determinants of the quality of part-time and full-time employment. It may also be useful to look at the proportion of addressees Evaluating impacts on employment working for temporary employment agencies, the rate of unionisation in firms where jobs are created and make comparisons of wages in respect to sectoral / industrial and regional averages. Because numerous measures co-financed by the Structural Funds have only a long-term effect on employment, apart from the temporary jobs which are needed to implement them in the first place, results and intermediate or specific impact indicators tend not to reflect employment effects. This is par­ ticularly the case of interventions in the areas of infrastructure and produc­ tive environment. For these types of intervention the more immediate non- employment impacts can be converted into predicted employment effects. For example, by dividing the turnover resulting from the intervention, by the average additional turnover needed to create a full-time job, it is possible to estimate the maximum level of jobs indirectly created. This method is useful in ex ante evaluation to quantify the expected effects of interventions. It may also be useful in intermediate evaluation to assess indirect long-term employment effects from established intermediate impacts. The conversion coefficients used in such an exercise are determined in relation to previous experience, research or econometric models. Whenever such coefficients are used, the basis of their definition should be clearly indicated. When interventions in the area of human resources focus on maintaining or improving the employability of individuals, employment impacts occur much earlier in the causal chain. In this case the choice of indicators depends on the definition of employability. An operational approach is to associate employability with the maintenance of employment of individuals already in work or the recruitment of individuals out of work. When studying the maintenance of employment of existing workers, it is necessary not only to take into account the fact that employment may be maintained within a firm or sector (in the case of an industry level training programme), but also that it may be maintained in a wider inter-firm or inter- sector perspective (i. e. employment maintained at a comparable level of quali­ fication outside the firm or industry under consideration). The wider the pers­ pective of employment maintenance, the more problematical its evaluation since the development of an individual's situation is more difficult to monitor outside a firm or area of activity. Furthermore, the links between the level of qualifications, jobs and salaries can vary greatly, making comparison difficult. The in-depth analysis of employment impacts also requires an analysis of confounding factors, i. e. those factors over which the programme has no control and which have an influence - whether positive or negative - on Transversal evaluation of impacts on intervention priorities employment or unemployment. The main effects of confounding factors can be identified by means of indicators. Indicators of adverse effects on employment may also be established (e. g. number of redundancies in the agricultural sector due to rationalisation of farming enterprises). These serve as warnings, or to enhance knowledge about the harmful effects on employment of the measures for which there is no stated employment objective. Indicators may also be used to make a judgement in terms of efficiency, in which case they relate result or impact indicators to the resources used, producing a cost / effectiveness ratio which may be compared to reference data. Examples of this type of indicator include the average cost per job created or maintained. • Precautions to take when using and interpreting indicators of employment effects The accurate evaluation of employment effects requires that the time needed for the effect to be produced should be taken into account, so as to avoid producing invalid information which might be based on the under-esti- mation of expected effects (i. e. not taking full account of effects which will subsequently occur) and / or their over-estimation (i. e. not taking account of the limited duration of temporary effects). In the case of intermediate evaluations, employment indicators are meaning­ ful above all for counting the number of temporary jobs created or maintai­ ned by the implementation. Impact indicators must take account of the delay that occurs before employment impacts are produced, which varies depending on the type of intervention being studied. A single indicator may be used to measure several levels of effects if used at different points in time. In the case of an intervention aimed at attracting new businesses to the area under redevelopment, the number of jobs created or maintained might include: an indicator of the jobs required for the develop­ ment being undertaken to attract businesses to the area, an indicator of the result of the intervention if it measures the number of jobs provided by firms when they set up, and an impact indicator if it counts the number of direct and indirect jobs created in the area. Finally, when comparing information provided by an indicator, it is important to ensure that the comparison is relevant. For example, a comparison on the Evaluating impacts on employment basis of "cost per job created" is only relevant if the context of the reference situation is the same. By way of an example, a system of indicators that could be used to track the production of employment impacts generated by a training programme for the unemployed is presented in Box 24. Box 24 - Example of a system of indicators for a return-to-work programme Programme Indicators Context Indicators (Related to the Intervention and its effects) (Related to the assisted area) Output Indicators Number of addressees Result indicators Rate of Immediate transition Impact indicators Gross placement rate Net placement rate Quality of jobs Number of long term unemployed participating In the programme (of which women) Number of addressees who are either hired or In a better social situation at the end of their participation in the programme (of which women) Number of addressees hired after 12 months (of which women) Number of long-term unemployed in the region Gross rate less estimated deadweight (trainees who would have been hired without assistance) less unacceptable substitution (trainees whose hiring has led another worker to long-term unemployment) % of hired addressees (of which women) who's job Is full-time and permanent after 12 months % of jobs In the region that are full-time and permanent Source: Volume 2 of the MEANS Collection ■ Sources of useful information Numerous series of statistics have been established at the European, national and regional levels. At the Community level, a Labour Force Survey is conducted annually throughout the Union. The current series of surveys was launched in 1983 and a survey is carried out annually. The structure of the survey is such that the data presented is disaggregated by gender, allowing the equal opportunities dimension of employment effects to be integrated into an evaluation. Many other data are also available through this survey such as: educational levels, number of hours worked per work, the presence of young children, etc. Transversal evaluation of impacts on intervention priorities Category Disaggregation by : Unemployment Jobs • Gender • Age group • Gender • Professional status • Field of activity Groups of job seekers • Gender • Duration of unemployment • Mode of job-seeking The Regio regional database covers unemployment and labour force surveys, and the results are disaggregated by region. Statistics are available on total employment, salaried employment, active population, long-term unemployment and remuneration of employees. Data are available from series which go back to 1970, 1975 or 1983. The reporting period is annual, with the exception of certain unemployment statistics which are monthly3. The statistical observatories MISEP (Mutual Information System on Employ­ ment Policies) and SYSDEM (System of Documentation on Employment) provide information on the different national employment situations. Among the drawbacks of using statistical surveys is the problem of non-cor­ respondence between place of work and place of residence which may create particular difficulties in the interpretation of statistics when an analysis is focused on unemployment. For example, due to commuting, the jobs recorded in a region are not all occupied by residents of that region. Statistics may therefore indicate a falling trend in unemployment levels, without the majority of the addressees of that improvement being residents of the supported area targeted by the intervention. This phenomenon is par­ ticularly important when the assisted area is small (e. g. urban areas in indus­ trial decline that are eligible for support to promote reconversion) or when border regions are concerned. Permanent statistical surveys on employment are also available from national statistics institutes, labour and employment administrations (statis­ tics on unemployment databases on human resources, etc. ). However, a dif­ ficulty with this sort of data is the time delay between the collection of data, its processing and eventual publication. Databases such as those compiled for the LEADER Community Initiative have proved to be equally useful. The LEADER Observatory gathers data 3. Labour force survey and data from the Regio data base can be obtained from Eurostat. Evaluating impacts on employment concerning the objectives of projects (including employment) and elements concerning counterfactual situations. There has been a surge of interest over the last few years in evaluating employment effects and evaluations in this field are rapidly evolving. For this reason an inventory of existing research reports and previous evaluations represents a valuable stock of knowledge on which evaluations can be built. The employment data provided by the evaluations of a programming cycle provide an extremely useful basis for estimations of effects of the next cycle. Estimations of employment effects which feature in programmes and ex ante evaluations should also be used in intermediate and ex post evaluations. Information produced by a monitoring system includes data that are necessary to evaluate employment effects. When the monitoring system comprises an entire set of indicators on resources and on physical and financial outputs, the extent to which the operational objectives of interven­ tions are being progressively attained and the associated employment of human resources during the implementation of interventions can be shown. However, it is important to bear in mind that monitoring can only, at the most, focus on addressees and would­be addressees (i. e. rejected applicants). This means that an analysis which focuses uniquely on monitoring data ­ for example, the construction of the counterfactual situation ­ will be biased. Furthermore, data gathering which is restricted to addressees and would­be addressees will be the subject of respondent bias. Nonetheless, monitoring data can be a useful source of information, since monitoring systems of most types of intervention include some information on employment. Examples of indicators that can be directly informed from the monitoring system include the employment generated by businesses attracted into the target area and the number of participants in a training programme who have obtained a qualification. ■ Appropriate tools for analysing employment effects An assessment of the impact of interventions on employment can be underta­ ken using the "step­by­step" approach presented in Part IV of this Volume. For an in­depth evaluation of employment effects, a wide variety of evaluation tools can be employed for the analysis of employment impacts. While these tools are extensively dealt with in Volume 3 of the MEANS Collection, a number of examples, drawn from across the spectrum of evaluation expe­ rience in this theme, are presented in this section. Transversal evaluation of impacts on intervention priorities Tools for in-depth analysis of employment impacts can be divided into four categories: • descriptive tools that allow for the treatment of both qualitative and quanti­ tative information (e. g. full case-studies, including longitudinal studies, and surveys), • comparative tools based on observation (either between similar cases or between the subject case and a randomised control case), • analytical tools that focus only on the situation being analysed (e. g. shift- share analysis), • models which are built on the basis of economic analysis of other expe­ riences and simulations of varying degrees of complexity (e. g. multipliers and coefficients, input-output and econometric models which combine a number of coefficients). Tools for quantitative analysis, such as shift-share analysis, do not only need to be used with skill, their limitations also need to be acknowledged. The measurement of their employment effects needs to be interpreted in a manner which takes account of qualitative factors. Both the level and scale of the intervention being evaluated have a number of implications for the type of tool most likely to be appropriate for the mea­ surement of employment effects. The investment of resources needed to produce dependable simulations means that full econometric models, such as the HERMIN macroeconomic model, are generally available for only higher-level, larger-scale interventions which operate at the national level or across very large regions. Accordingly, such approaches are more likely to be of most direct use in the evaluation of the effects of programmes designed to promote the development and structural adjustment of lagging regions, or in the evaluation of the overall effect of the variety of activities typically funded by the Structural Fund on a national level. On the other hand, simple coefficients are rarely robust enough to transfer between situations which include a large number of variables. Ideally their use should be confined to very similar interventions with relatively direct employment effects (e. g. those which are targeted on similar groups in similar and smaller regions undergoing industrial reconversion or rural zones in the process of development and structural adjustment). Evaluating impacts on employment The same limitations apply to comparative analysis tools, since the more complex a situation becomes, the more difficult it is to find other situations against which comparisons can be made. They are, however, particularly useful for establishing the policy-off scenario of small-scale human resource interventions. Job creation or maintenance effects lend themselves to analysis by a variety of top-down approaches (i. e. relatively sophisticated statistical techniques, modelling tools, etc. ) and bottom-up approaches (i. e. surveys, case studies etc. ). As for interventions targeted at employability effects, these are relati­ vely small and should therefore be analysed using the bottom-up approach. • Surveys and case studies: Evaluating Training and Enterprise Council (TEC) services (United Kingdom) This evaluation sought to assess the overall cost-effectiveness of a programme of business counselling, skills training and information and advice for existing small and medium sized enterprises (SMEs). Additional employment was only one of a number of objectives of the programme and the evaluation also took increased output, improved productivity, enhanced capital expenditure and profitability into account. With regard to employ­ ment effects, the objectives of the evaluation were primarily to assess dis­ placement and to explore the possibility of any crowding out effects. The evaluation used: • a postal questionnaire sent to the management of 75 English TECs (concerning their activities, targeting, etc. ); • personal interviews with 20 TECs to provide qualitative information on the inputs to their activities (e. g. strategy developed, benefits resulting from different service approaches, etc. ); • six case studies mainly focused on customer benefits and on the companies' own views on the impact of the services on jobs and output. The questionnaire was designed to distinguish between gross and net benefits. Following the conventional path to identify the net impacts on employment, the study attempted to assess (gross) effects, deadweight and displacement effects. All of the results of the case studies on the local economy were aggregated to the national level and net benefits on the local level were carefully distinguished from the net impact at national level. To do this, it was necessary to take into account national displacement as well as Transversal evaluation of impacts on intervention priorities macro-economic crowding-out effects since gains in one TEC area may lead to displacement in another area, and if all areas show increased economic activities, benefits are likely to be crowded out (e. g. by higher inflation rates). • Comparative analysis: Measuring the effects of Swedish employment training on income and employment (Sweden) 1995 Two experimental groups consisting of 4,000 trainees who completed their training in 1992 and 6,000 who completed it in 1994, were included in the survey. A very high proportion of these were individually matched with a control group which received no training at the same time as the subjects. Some policy changes had taken place between the training of the two cohorts, so a detailed study of these particular changes was also possible. The analysts used a fairly sophisticated control function (including values for background variables and error) to trace the relationship between training and income. The findings obtained showed significantly different results for the two years, with the 1994 trainees having significantly higher earnings than those who had not received training, but the 1992 trainees experiencing no significant advantage. For those who ended up working in the vocation for which they trained, earnings were higher than for the control group. In order to measure employment effects, regression analysis was used. The findings reported showed that while the control group had found it increasin­ gly difficult to find work during the period when the subject group was training, those who had subsequently been trained found it no more difficult to obtain work. Thus, even if their position had not improved in absolute terms, it was relatively better than it would otherwise have otherwise been. • Statistical analysis: An ex post evaluation of the integrated action programme for Limburg using shift-share analysis (Belgium) The goal of the programme being analysed was to support job creation in new activities to compensate for the decline of employment in the mining industry. The evaluators tracked employment change during the 1980s and early 1990s on the basis of sectoral and other characteristics for both Limburg and Belgium as a whole. Shift-share analysis was then used to estimate the number of jobs created by the programme in comparison with what would have happened in its absence, on the assumption that the rate of decline in the traditional sectors was equal to the Belgian average. Evaluating impacts on employment The relatively positive experience of the Limburg region (high levels of economic growth, new firm creation and employment) was attributed primarily to export growth in one or two sectors and exceptional growth in the number of small enterprises. Re-employment patterns amongst ex- miners were put under particular scrutiny in light of the objectives of the intervention, as were the investment instruments chosen. On the basis of this analysis, specific proposals were made for a modification of these ins­ truments to increase employment effects in subsequent interventions. • Multipliers and coefficients: Evaluating labour support schemes using standardised coefficients (France) This approach is used for the ex-post and on-going evaluation of effects on employment and unemployment of labour support schemes financed by the French Government. It takes into account both deadweight and attraction effects. For each single measure a coefficient of deadweight and a coefficient of attraction are calculated, based on the evidence of past evaluations and on intuitive adjustments. These coefficients are then applied to the variations of the stock of addressees of each measure, over a given period of time, in order to obtain the effects on employment and unemployment of each measure. The effects on employment correspond to the difference between the variation in the stock of addressees and the quota of these addressees (obtained by applying the coefficient of substitution) who would have obtained a job in any event, or who substitute for another worker. Calculations of effects on unemployment also take into account the coeffi­ cient of attraction and the absolute value of the reduction in unemployment resulting from the difference between employment effect and the quota of new addressees attracted to work by the measure. This macroeconomic approach is especially valuable since it allows the effects of cyclical changes to be expressed as changes in the coefficients over time. It is globally useful as it gives a complete view of existing labour support schemes and of the net effects of each one on employment and unemployment. However, it has a limited time horizon and does not provide much information about job tenure and, in general, about the quality of jobs. The method is easy to apply once the coefficients have been obtained, but establishing the coefficients in the first place demands a high level of infor­ mation input. Transversal evaluation of impacts on intervention priorities • Input-output analysis: Evaluating an objective 1 Operational Programme in Italy The tool used for this exercise is a Social Accounting Matrix (SAM), which integrates Keynesian income multiplier mechanisms into a standard input- output model. In a SAM the various interdependent links of an economy are quantified and simulated. The impact mechanism relies on global multipliers, obtained from a wider matrix system, which add additional rounds to the input-output mechanism. The global multipliers are able to capture effects arising not only from the relations linking the productive sectors, but also from other relations linking economic actors. The increase in factor revenues (value-added) generated by input-output mechanisms is translated into an increase in income. The level of consump­ tion then rises (according to the average consumption rate), creating an increase in final demand which corresponds to a new impulse to the economy and to a new activation of the process. Effects on employment are calculated on the basis of a hypothesis on the evolution of productivity by sector, which is related to the changes in value added obtained from the input-output application. The reference model for this exercise is a two-region (Campania and the rest of Italy) SAM. The regional SAM tables are obtained from the 1988 input- output table for Italy, which is split using the results of two surveys on the economy of Southern Italy: one on the purchase of intermediate goods by firms from this area and the other on income and consumption of house­ holds. The SAM used has a low level of disaggregation: two factors (labour and capital) and two types of actor (firms and households). With regard to sector disaggregation, two versions are provided: one with four sectors (agriculture, industry, building industry and services) and another with sixteen (with industry and services split into sub-sectors). The model is applied in two steps, each with different procedures, in order to separately capture demand-side and supply-side effects. • Macroeconomic modelling: The HERMIN model in Ireland The macro-economic model HERMIN has been developed by national teams in four objective 1 countries: Ireland, where it was initially developed, Portugal, Greece and Spain. This project is aimed at creating a system of harmonised econometric models for the evaluation of the medium-term economic performance of the European economies. Evaluating impacts on employment As econometric models go, the Irish version of HERMIN is relatively small with 50 equations, about 150 identities and 60 exogenous variables. The standard version of HERMIN has four sectors: tradable (manufacturing industry), non­tradable (market services, building and utilities), agriculture and public. In the two main productive sectors (tradable and non tradable), simple production functions are used, while the agricultural sector has not been modelled because of the exogenous factors related to the CAP, which influence aggregate farming behaviour. The public sector is policy­driven, with expenditure and tax rates set as policy instruments, subject to budget constraints and borrowing sustainability. The consumption function of the model (based on Keynesian multiplier mechanisms) does not play an exclusive role in driving this model as it does in demand­side focused models. In this case the supply­side features of the industrial sector are also influential. From the production function, factor demands are derived in both productive sectors, on the basis of cost minimi­ sation subject to given levels of output. Levels of employment and invest­ ment are then dependent on the level of output, which is demand­driven in both sectors. In the tradable sector output is driven by world demand and competitiveness. In the non­tradable sector, on the other hand, actual output is driven by a weighted measure of final demand. The supply of labour is modelled in order to take into account migration flows, which are fairly significant in Ireland. The work­force participation rate is influenced by migration, unemployment (the discouraged worker effect) and the replacement ratio (i. e. the fraction of average earnings replaced by social welfare transfers). ■ Summary The creation of employment effects is an objective in all areas of interven­ tion, and the capacity to create or maintain employment is an important factor in selecting projects for support. A second type of employment effect is sought by interventions in the area of human resources, primarily in training. In this case it is linked to the maintenance or improvement of the employability of individuals either in work or unemployed. In addition to the primary difference of job creation or maintenance effects and employability effects, employment effects vary widely. They occur at different points of the causal chain, they can be temporary or of a more long­ Transversal evaluation of impacts on intervention priorities term nature, they can be sustainable or transitory. For these reasons the aggregation of different types of employment effects is often not possible. When estimating net employment impacts, nature of deadweight, displace­ ment and substitution effects vary according to whether job creation or maintenance effects or employability effects are being evaluated. A great deal of experience has been accumulated in this area of evaluation and a wide range of bottom­up and top­down tools have been successfully used to evaluate employment effects. ■ For m o re information see. Bradley J. , Whelan K. , Wright J. (1993), Stabilization and Growth in the EC Periphery, Aldershot: Avebury. Department of the Environment. (1988), An Evaluation of the Urban Develop­ ment Grant Programme, London: HMSO. Gautié J. et al. (1994), Les subventions à l'emploi ­ Analyse et expériences européennes, Cahier travail et emploi, La Documentation Française. Issaev B. , Njikamp P. , Rietveld P. , Snickars F. (1982), Multiregional Economie Modelling: Practice and Prospects, North Holland. King J. (1990), Regional Selective Assistance 1985­88: An Evaluation by the Department of Trade and Industry, the Industry Department for Scotland and the Welsh Office Industry Department, London: HMSO. Manski CF. , Garfinkel I. (eds. ) (1992), Evaluating Welfare and Social Pro­ grammes, Harvard University Press. Njikamp P. , Rietveld P. , Snickars F (1986), Handbook of Regional and Urban Economics, North Holland. OECD. (1991 ), Evaluating Labour Market and Social Programmes ­ The state of a complex art, Paris: OECD. OECD. (1998), Employment Outlook, Paris: OECD. Schmid G. , O'Reilly C, Schöman K. (eds) (1996), International Handbook of Labour Market Economics, Cheltenham: Edward Elgar. Storey D. J. (1990), "Evaluation of Policies and Measures to Create Local Employment", Urban Studies, vol. 27, n°5. Evaluating the impact of structural interventions on equal opportunities ■ Why do impacts on equal opportunities need to be evaluated? • The Community context The European Council has declared on several occasions (in Essen, Cannes and Madrid) that the promotion of the principle of equal opportunities between men and women is, along with the struggle against unemployment, a fundamentally important task facing the European Union and Member States. Glaring inequalities continue to exist between men and women in all areas. As far as the participation of women in the job market is concerned, their level of activity is lower than that of men in all the Member States of the European Union. In 1996, the employment of women over the age of 15 varied between 34,6% and 58,7% (Eurostat). The wage gap remains signifi­ cant in all countries. Furthermore, women are generally more affected by unemployment than men in all age groups, particularly as regards long­term unemployment. Finally, most of the actions of the structural policies have dif­ ferentiated effects, reproducing discrimination between men and women. Efforts to promote equal opportunities at European level are not intended merely to introduce specific measures in favour of women, but to add a dimension to all actions and structural policies. This concept is known as "mainstreaming" (see Box 2). Mainstreaming is intended to Influence the strategies of Structural Funds policies so that they incorporate the dimension of equal opportunities" between men and women by systemati­ cally examining their respective situations. Policies must therefore be defined by taking into account the diversity of needs, including the needs of the various groups of men and women in society. The "Fourth Action Programme on Equal Opportunities for Women and Men (1996­2000)" is aimed at integrating equal opportunities into the preparation, implementation and monitoring of all policies and actions of the European Union and Member States, with due respect to their respective competencies. With regard to the Structural Funds, the most noteworthy event was the revision of the framework­regulation in 1993, in which equal opportunities are explicitly referred to, with particular emphasis on the struggle against unemployment among women. In 1996 the Council adopted a Resolution Transversal evaluation of impacts on intervention priorities reaffirming its political will in this respect. In decision-making concerning the approval of programmes, a standard clause was specifically introduced to guarantee compliance with this new obligation. Several examples of good practice exist, taken from the successful results of the NOW 1991-94 (New Opportunities for Women) experience, and covering a broad sample of measures. NOW, the first Community Initiative specifically for women (focused on the promotion of equal opportunities in the job market), has opened new avenues to ensure that women are taken into account more fully by all the Structural Funds. From the beginning of the new period of Structural Funds programming (1994-1999), the principle of equal opportunities has been considered as an objective of structural policies in their entirety. There has thus been a shift from a passive conception (respect for the principle) to an active conception (objective of all structural policies) of equal opportunities. Today, equal opportunities are no longer considered only to be the problem of women. The promotion of equal opportunities is recognised for its ability to contribute towards socio-economic development in general and regional readjustments in particular. In the context of Structural Fund interventions, the evaluation of the applica­ tion of the principle of equal opportunities involves assessing the potential or actual impact on the respective situations of men and women and the extent to which equality is taken into account in processes and mechanisms of pro­ gramming, implementation, monitoring and evaluation. This latter aspect involves, for example, analysing of the role of women in interventions, measuring positive developments and highlighting discrimination. • The need to assess the application of the principle of equal opportunities Evaluation can play an essential role as a mainstreaming tool by ensuring that the principle of equal opportunities is taken into account in structural policies. It may, in particular, help to enhance the effects of synergy for all the actions benefiting from Structural Funds support. In order to do so, it is necessary to apply techniques which take into account the specific nature of equal oppor­ tunities between men and women. Through the definition of evaluation criteria relating to this theme, and of corresponding indicators, evaluation can be used to systematically examine these actions and policies and to Evaluating impacts on equal opportunities take into consideration their direct or indirect effects in terms of equal oppor­ tunities. Methodological difficulties are not as great in the assessment of the impact of specific actions in favour of women (who are, in this case, the direct bene­ ficiaries) as in the transversal evaluation of the effects of Structural Fund pro­ grammes in terms of equal opportunities. Interventions likely to produce direct effects on women are those focused on the development of human resources. Other types of intervention concerning infrastructure or the pro­ ductive environment do not necessarily include equal opportunities among their explicit objectives. Their effects are therefore more likely to be indirect. In the context of the evaluation of Structural Funds, a few isolated initiatives have been taken to evaluate the promotion of principle of equal opportuni­ ties. This experience has, however, been limited to a few cases of ex ante and intermediate evaluation. Ex post evaluation experience in the Structural Funds context is still embryonic. The NOW Community Initiative which was evaluated ex post is too specific to serve as a model in the framework of classical operational programmes. ■ Defining what needs to be measured It is not possible to define ex nihilo what an evaluation has to assess, in so far as the strategy chosen to promote equal opportunities depends to a large extent on national and even regional contexts. Thus, the application of the principle of equal opportunities varies considerably from one Member State to the next. For certain countries (France, Portugal), it seems to be a general objective but without indications as to the strategy to adopt or actions to implement. In Spain and Germany equal opportunities are taken into consi­ deration in the context of specific actions, and are promoted by means of general measures. In Finland, the principle is visibly and thoroughly integra­ ted into all the components of programmes. • Analysing the process of programming interventions Because the programming of structural interventions is the result of decisions taken jointly (by the EU, state, regional and local authorities), it is necessary to examine whether, at all levels, there is a sufficiently strong political will to take into account the interests of women and to meet their specific needs. Keen awareness of the question is essential if suitable solutions are to be conceived and implemented (e. g. the setting up of Transversal evaluation of impacts on intervention priorities actions to inform and train the key actors). The degree of awareness is not only related to the level of representation of women in decision-making bodies, even if the presence of spokespersons is a contributing factor. Example: Training in occupations traditionally reserved for men. Training for women in traditionally masculine occupations has frequently been unsuc­ cessful. Women are not necessarily recruited by firms in these sectors, or for jobs for which they have been trained. Sometimes they are recruited but give up their jobs due to pressure from an exclusively masculine environment. An ex ante evaluation can help to reduce this failure rate by programming, in addition to the training measure, actions of the following kind: awareness in firms, promotion of recruitment of women, advice and assistance to women after recruitment. This example and others presented in this chapter, are inspired by real situa­ tions. In some cases, however, they have been modified in order to provide a concise, but effective, illustration of an idea or a concept. that the resources allocated to promoting equal opportunities are The level of analysis in the example is upstream from the implementation of the programme and thus of the production of effects on equal opportunities. The evaluator must examine the conditions in which the programmes were designed and check: • whether the announced objectives take into account the interests and specific needs of women; • whether groups of women and / or authorities responsible for equal oppor­ tunities at national and regional level have been involved in the programming; • whether the strategy and choice of the modes of intervention have relied upon a prior assessment of women's needs and potentialities; • coherent with stated objectives; • the proportion of interventions programmed that are able to facilitate the achievement of priority objectives concerning equal opportunities; • whether the programming documents (CSF, SPD, etc. ) provide qualitative and quantitative criteria and precise indicators for assessing the considera­ tion given to equal opportunities; • whether some measures are likely to run counter to the principle of equal opportunities, risking the worsening of the existing imbalance between men and women; • whether the SPD is consistent with national and regional equal opportuni­ ties policies; • the proportion of women in decision-making bodies. Evaluating impacts on equal opportunities • Analysing the conditions in which interventions are implemented The perception of women's interests is not always the same during the pro­ gramming of interventions and during their implementation. It is essential to analyse the conditions of implementation in order to understand how they can account for the achievement or not of equal opportunity objectives. Example: Organisation of training for long-term unemployed Presuming the monitoring of training measures revealed that the drop-out rate of women was high, an intermediate evaluation would have made it possible to examine the causes, e. g. family reasons (spouse's hostility), lack of infrastructure (lack of child- care facilities, means of transport, care facilities for an aged or ill parents, etc. ) or financial reasons (the allowance is too low to cover child minding costs, etc. ). On the basis of such conclusions it becomes possible for those responsible for the running of the scheme to take appropriate steps to remedy the situation. To ensure that the principle of equal opportunities is taken into account in the implementation of structural interventions, the evaluator must check: • whether the selection of projects has included the application of the principle of equal opportunities as a condition in allocating budgets; • whether groups of women and / or authorities responsible for equal oppor­ tunities are represented and active in the monitoring committees; • whether the people responsible for programme management are aware of all issues involved in promoting equal opportunities (e. g. have awareness campaigns on the subject taken place); • whether positive discrimination (sometimes referred to by the term "affir­ mative action", i. e. the aiming of measures at a particular group with the objective of eliminating and preventing or compensating for disadvantages which are a result of existing attitudes, behaviour and structures) has been applied to favour the participation of women in project initiation (e. g. whether there are women's NGOs among the promoters); • whether monitoring systems include the collection of gender differentiated data; • whether the interventions have been improved during their implementa­ tion, to better meet women's needs; etc. • Highlighting the effects of interventions on equal opportunities The main methodological difficulty encountered in this type of arises from the fact that two concomitant priorities have to be evaluated: economic and social cohesion on the one hand and equal opportunities on the other. The latter is rarely formulated explicitly as an intermediate objective of structural interventions, except in the case of specific measures or priorities. Moreover, Transversal evaluation of impacts on intervention priorities structural interventions are rarely, except for those of the ESF, aimed at people as direct beneficiaries. The effects of structural interventions are usually indirect and long term, which makes their evaluation complex. Application of the principle of equal opportunities to all interventions is a transversal objective that cannot be analysed immediately in its globality. It is therefore advisable to break down the effects into coherent sub­sets. The following typology is intended to provide an analytical grid, grouping together four types of possible effects in the specific context of Structural Funds, in terms of: • improving living conditions so as to better meet women's needs; • greater accessibility to the job market for women; • improving the situation of women at work; • promoting of the participation of women in the creation of socio­economic activities. Box 25 presents an objectives tree that draws on this typology to illustrate how this tool can be used to structure the mainly implicit objectives of actions promoting equal opportunities. Actions such as support for further or conti­ nuous education, professional training, counselling and orientation can all increase the employability of individuals. The introduction of more flexible work hours for women with children or aged relatives at home, leave organised in conjunction with school holidays, reduction in the physical aspect of jobs, Box 25 ­ Example of a thematic objectives tree for the priority of equal opportunities Global objectives Examples of specific objectives Balanced development for men and women Living conditions which meet women's needs ι ­ ­ • Improved access to neighbourhood services • Improved access to specialised medical treatment • Adaptation of transport systems Y Access for women to the job market • Restore employability • Adapt available jobs • Improving skills Women's situation at work « ­ • Reducing professional segregation • Improving working conditions Participation of women ■*■ ­ so cio ­economic activities • Favour the creation of business or independent activities by women Evaluating impacts on equal opportunities can all contribute to adapting jobs to women's needs. As a consequence of these actions, access for women to the job market can be improved. Furthermore the different intermediate objectives are not unrelated. If the living conditions of women were improved so that needs could be met more ade­ quately, the other three types of impact would more easily be achieved. Similarly, improvements to the situation of women in their jobs may improve conditions of access to the job market. It is for this reason that, the links between intermediate objectives in the thematic objectives tree have been shown. In order to facilitate the work of evaluation teams, many examples of effects are classified in terms of this typology of impacts and presented in Box 26. Transversal evaluation of impacts on intervention priorities Box 26 - Example of effects by category of impact Category Lifestyle Access for women to the job market Examples of effects • Relaxing constraints related to responsibility for small children or for other persons, either temporarily (e. g. during job seeking) or over a longer period (e. g. investing In a middle-term social or professional project) • Improving integration and reinforcing the capacity for initiative of women in rural areas, by providing more adequate public transport • Reconciling family and professional constraints more satisfactorily • More assistance in dealing with social problems, particularly through improved social services (quality family planning, support for victims of household violence, sexual harassment, etc. ) • Increased access to good quality care services for children and other depen­ dants • Promotion of gender equality In parental leave • More flexible work conditions • Less sectoral and vertical segregation between men and women in education and training • Improvement of job prospects for young women (guidance towards occupa­ tions with a future) • Enhanced employability of women • Provision of jobs which are better suited to meet women's requirements • Adjustment of the content of training actions to meet the particular requirements of women (training in fields that improve their career prospects), and adjustment of the practical modalities of training to cater for women's specific constraints such as their daily schedule • Personalised aid for the unemployed In finding a job. Including more appropriate support for women (e. g. for women trying to re-enter the job market) • Setting up professional alternatives for women working in the underground economy, etc. • Reconciling family and professional constraints more satisfactorily Women's situation at work • Reduction of sectoral segregation by promoting a balanced representation of women and men in all economic sectors • Jobs better suited to qualifications • Increasing the proportion of women in senior positions • better access to ongoing education, etc • Reconciling family and professional constraints more satisfactorily Participation of women In the creation of new activities • Better recognition and appreciation of the role of women in the creation of new businesses • Improvement of women's access to services and infrastructures supporting the creation of enterprise, and to financial facilities for business development • Greater participation of women in local development initiatives and in new sources of employment, etc. • Reconciling family and professional constraints more satisfactorily Evaluating impacts on equal opportunities It is also possible that interventions, no matter their effectiveness and success, contribute to reproducing existing discriminatory patterns, thus having unfavourable or undesirable effects. It is very important that the evaluator analyses the over or under­representation of women in particular interventions and that the findings are contextualised. Women may for example be over­represented in measures not directly or immediately leading to employment, e. g. training measures, as opposed to apprentice­ ship schemes or unemployed aids. The importance of providing an explana­ tory socio­economic context to the findings is illustrated by the issue of part­ time work where the occurrence of such work can be considered as positive or negative depending, for example, on whether the part­time work was chosen or necessitated and taking in account the frequently less favourable working conditions, both in terms of work security and wages. When pro­ grammes are orientated towards traditionally male­dominated sectors, the evaluator could, in addition to determining to what extent women have parti­ cipated in measures, which types of measure and results, try to identify the conditions and necessary changes in these needed to promote increased gender balance in the sector in the future. ■ Examples of impacts by area of intervention The approach used to integrate equal opportunities into programming documents should be twofold: an approach providing for measures expli­ citly in favour of women, and a transversal approach considering equal opportunities as a general guideline applicable to all the measures proposed. In the latter case, the promotion of equal opportunities is usually simply mentioned as a subsidiary of the general measures. In view of the absence of an operational translation of this priority, as well as its marginal nature, it is essential to examine how general measures incorporated into classical Structural Fund programmes can potentially contribute to equal opportunities. Transversal evaluation of impacts on intervention priorities • Basic infrastructures Among the interventions in the field of infrastructure, those which seem to incorporate an equal opportunities dimension concern basic infrastructure and facilities of a social nature. Example: Child c a re facilities in Portugal With s u p p o rt f r om the N OW Initiative, a p l a y s c h o ol w as created In the heart of the industrial area of Evora (Portugal), c l o se to an o c c u p a t i o n al training c e n t r e. Child m i n d i ng facilities were m a de available to jobless w o m en wanting to a t t e nd training c o u r s e s, a nd to e m p l o y e es of f i r ms in t he Industrial area. T he p l a y s c h o ol w as e q u i p p ed with a playground and open-air s w i m m i ng pool in order to make the setting m o re homely, In w h at is an industrial a nd essentially m a s c u l i ne e n v i r o n m e n t. T he c e n t re t h at h ad h i t h e r to o f f e r ed t r a i n i ng in t r a d i t i o n a l ly m a s c u l i ne t r a d es (e. g. carpentry), had been frequented mainly by m e n. Today, 4 0% of the participants are w o m e n. This centre has enabled s o me of t h em to find j o bs and others to find more s e c u re e m p l o y m e nt t h an t h ey f o r m e r ly e n j o y e d. T he c h i ld m i n d i ng s e r v i c es has helped to ease the family constraints of w o m en w ho consequently have the possibility of involving themselves in m e d i u m - t e rm professional projects. Outputs > Child care facilities Use of facilities for lightening family constraints Increased availability Effects Better access to training Qualifications improved and/or widened Successful projects O Better access to job market Improved jobs situation Enhanced participation in economic activities Priority I I Evaluating impacts on equal opportunities • Human resources The principle of equal opportunities is more visibly incorporated into measures concerning human resources. The most significant effects are in activities involving consultancy, guidance and monitoring; actions in favour of training and occupational integration; interventions in education systems, training and employment; and, lastly, on work organisation and modes of production. Example: Setting up measures for training, follow-up and rehabilitation tailored to suit individual requirements (Portugal). Trainees with children or adults under their responsibility were given priority in training actions and in the allocation of additional benefits. This enabled both w o m en and men to reconcile their family and working lives. The training also developed methodologies and specific strategies intended for w o m e n, in the framework of measures providing assistance and rehabilitation support (individual counselling, j ob seeking techniques, career p l a n n i n g, etc. ). C o n s e q u e n t ly a more b a l a n c ed p a r t i c i p a t i on of m en a nd w o m en in all sectors of activity was possible. Outputs > Financial support for people with children or adults In their care System for providing support Individual rehabilitation course Effects Better reconci­ liation of family and working life Easy access to training Improved and diversified qualifications More balanced participation in all sectors of activity Information Counselling Continuous support Quality of training systems ΐ> Improved work situation Access to jobs Creation of economic activities Priority I I Transversal evaluation of impacts on intervention priorities • Productive environment Actions relating to the adaptation of productive activities and to local and regional development are likely to have an impact on the promotion of equal opportunities. Example: Creation of a European Business and Innovation Centre (Italy) A centre for the development of business innovation was created in the Pouilles region. The Business Information Centre in Puglia Sprind developed an international marketing methodology based on employing mainly women in the marketing division. Prospecting for business opportunities was based essentially on the ability of the women in the team to stimulate their interlocutors' curiosity and interest in their new products. Local products of SMEs were introduced into international distribution channels and the firms concerned enhanced their competitiveness on a global scale. This favoured the establishment of new SMEs in the region, specialising in agricultural products. The SMEs created jobs, most of which were occupied by women farmers living in and around the villages. Outputs > Creation for a development centre for innovative business in rural areas Establishment of other SMEs Creation of local activities Effects > Reduction of rural women's isolation Success of new enterprises created by women Access to employment 4 / 1 Creation of economic activities Priority ■ Defining and selecting indicators Volume 2 of the MEANS Collection is devoted to the subject of indicators for monitoring and evaluating Structural Fund interventions. The above Volume includes an information sheet devoted to the priority of equal opportunities. In order to remain as operational as possible, the use of indicators should be mainly limited to those which are quantifiable. Yet qualitative information must not be overlooked, given the complex and innovative nature of the objective of equal opportunities. The impacts of interventions on this priority are produced by an accumula­ tion of intermediate results derived from a series of heterogeneous outputs. Programme indicators should provide meaningful information on all the Evaluating impacts on equal opportunities steps in the process, ranging from the allocation of resources to the produc­ tion of impacts. A pragmatic selection should be made of a small number of ad hoc indica­ tors relating to equal opportunities. It should not be necessary to undertake costly collection of information, nor should an excess of information be supplied to decision-makers: • At European level, the priority need is to assess and clarify the conse­ quences of structural policies in relation to the extent to which equal oppor­ tunities are being taken into account. • At programme level, the monitoring committee needs to know to what extent resources have been consumed and to monitor outputs. When the programme has not included the principle of equal opportunities in its explicit objectives, the Monitoring Committee should be kept informed on the impact of the programme on men and women. • At project level, managers must monitor in detail the consumption of resources and the outputs produced, as well as the reactions of the different of beneficiary groups, particularly women. Women are not a homogeneous group, but constitute half of the population concerned by Structural Fund interventions. Because of this, the range of indicators should be systematically broken down according to gender. A system of indicators "sensitive to the question of equal opportunities" requires the close integration of types of beneficiaries and types of measures. This makes it possible, for example, to assess whether women benefit in the same proportions as men from measures with a direct effect on employment (such as recruitment or employment subsidies) or whether, on the contrary, they benefit only from training measures. Context indicators, by providing a baseline, show what proportion of the targeted population, particularly of women, was reached or to what extent the needs of women in the region were met. A programme indicator (e. g. the number of women beneficiaries of the programme who kept their jobs after having children) can be combined with a context indicator (e. g. the regional percentage of women between the ages of 25 and 35 who interrupted their careers after having children) to give an indication of the contribution of the programme to solving one of the problems faced by women in the region. No universal rule exists for constructing a system of indicators for the appli­ cation of the principle of equal opportunities. The quality of existing indica- Transversal evaluation of impacts on intervention priorities tors has to be checked to determine whether they are acceptable. The most frequent shortcomings are: • the absence of relations between the indicators and the targeted; • the difficulty of using the indicators due to the lack of reference data for the constitution of the counterfactual situation; • the absence of a sub-division between men and women in the indicators when referring to individuals. A system of pragmatic indicators taking into account the principle of equal opportunities can, however, limit the amount of information collected and transmitted according to its relevance for each managerial level. In the case of the ESF, indicators are usually presented as shown in the upper half of Box 27. When the principle of gender mainstreaming is applied, indicators are presented in the form shown in the table. Box 27 - Example of the application of the principle of gender mainstreaming Measure XX : Number of beneficiaries : • Total • by sex · by age · by level of qualification Gender mainstreaming implies the following breakdown of data: %Men % Women 1. Total 2. by age 3. by level of qualification • Selecting and interpreting indicators for a given programme Programming documents reveal an almost total absence of precise indica­ tors on how the principle of equal opportunities is being taken into account. As far as the participation of women in the CSF and SPDs are concerned, indicators are very general and very limited. At best, women are considered as a separate category of beneficiaries, like young or handicapped people. In contrast, more specialised monitoring bodies offer increasingly sophisti­ cated indicators. These are sometimes too general to be used in the evalua­ tion of a structural programme that does not have the promotion of equal opportunities as its main objective. An example is the "index of dissimilarity" which serves to analyse the degree of segregation between the sexes, by sector of activity or by profession. Evaluating impacts on equal opportunities Thus, the evaluation team will most often have to propose ex nihilo a list of indicators which incorporate the equal opportunities dimension into the global evaluation of a programme. Evaluators, however, cannot be expected to define specific indicators for the promotion of equal opportunities within this framework. It is therefore necessary to make a selection and to settle for relatively rough and approximate indicators to assess the equal opportuni­ ties dimension. Evaluation teams should use a solid analysis of the context based on a diagnosis of the problems facing women, the identification of prospects for development within the specific national or regional context or the identification of the most important aims within their region for the promotion of women. The interpretation of information provided by indicators is not always easy. It is essential to use "control variables" on the local, regional or national context, particularly to check whether the effect produced is truly beneficial for women. For example, an increase in part-time work for women could be explained by the lack of other professional alternatives. It could also reveal a lack of infrastructure, child-minding facilities and a persistent lack of sharing of family responsibilities. By way of an example, a system of indicators which could be used to track the extent to which the equal opportunities dimension has been taken into account within a programme of community centre regeneration is presented in Box 28. Box 28 - Example of a system of indicators for the establishment or regeneration of community centres Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Facilities Capacity Result indicators Satisfaction Impact indicators Employment Purchased, constructed or refurbished buildings in m! (of which: area devoted to meeting women's needs, i. e. nursery, etc. ) Maximum number of users that can be served (of which: maximum number of users of women-specific sources) % of user that are satisfied or very satisfied with the project (of which women/men) 1 Maximum number of users in nurseries in the region Number of users holding a permanent job (of which women/men) % of regional workers who use nursery services Source: Volume 2 of the MEANS Collection Transversal evaluation of impacts on intervention priorities ■ Suitable sources of information Data available from the Statistics Office is usually used to quantify context indicators. For example, many indicators can be found at Eurostat in the NEW CRONOS database, a collection of about 70 million harmonised items of data for the Member States of the European Union. The Regio database, devoted to statistics on economic life in each Member State, presents data in the form of about 70 standard tables classified by subject domain. The data are all broken down by region, according to the nomenclature of the NUTS territorial units, and cover a wide range of information: demography, unemployment, labour surveys, etc. Statistics compiled from labour surveys primarily concern the distribution of working hours, the breakdown of jobs by age, gender and socio­professio­ nal category, and the level of unemployment with a breakdown of figures in terms of the reason, duration and modalities of job­seeking peculiar to each country and sector of activity. A statistical portrait of women living in the EU in the early 1990s is presented in the EUROSTAT publication "Women and men in the European Union". The document includes opinion polls that enrich a highly detailed comparative analysis of women's social and profes­ sional situation compared to that of men. Furthermore, the national institutes of certain Member States compile national statistics on the subject "women". For example, in Spain the Women's Institute (Instituto de la mujer) has created over 100 indicators to analyse disparities between men and women in the following fields: demo­ graphy, education, employment, health, policy and social benefits. ■ S o me e x a m p l es of current practice in the European Union A detailed treatment of evaluation tools is provided in Volume 3 of the MEANS Collection. Some examples of the use of tools for assessing the extent to which the promotion of equal opportunities has been taken into account in Structural Fund interventions are presented in this section. The methodological singularity of this type of analysis lies in the complex and pervasive nature of the phenomena that has to be analysed. The construction of the evaluation must take into account the small number of former references and the lack of information available to date in this field. Evaluating impacts on equal opportunities An assessment of the impact of interventions on the priority of promoting equality between men and women can be undertaken using the "step-by- step" approach presented in Part IV of this Volume. For an in-depth evalua­ tion of impacts, however, other evaluation tools have to be employed. With the exception of the job market, for which numerous studies exist on discrimination against women within the OECD area as well as at national level, there are not sufficient well-established foundations or prior informa­ tion available for a "hypothetical-deductive" type approach, in other words, a procedure which assumes the formulation of precise hypotheses for struc­ turing the tools for data gathering and processing. The use of closed ques­ tionnaires, for example, must be limited to certain well-known situations such as a survey among trainees who have attended the same course. It is thus appropriate to opt for an "inductive" methodology suited to producing new knowledge about unfamiliar situations, based on the comparison of qualitative observations. In addition to the use of focus groups, other corres­ ponding techniques are unstructured interviews, full case studies or, more rarely, ethnographic observation. For ex ante or intermediate evaluations, the techniques developed in the field of corporate management (e. g. Ishikawa diagram, "I had a dream" and the scenario technique) can serve to diagnose problems and find innovative solutions. Descriptions of these tools can be found in most classic corporate management textbooks. Other suitable tools are SWOT (Strengths / Weak­ nesses, Opportunities / Threats) and colour vote. Ex post evaluation requires the use of a methodological "package" of different techniques - e. g. questionnaire surveys, full case studies and com­ parisons of several population samples at regional and national level - in order to give a credible picture of the real impact. • Ex ante analysis of projects financed by the ERDF in Spain In the framework of the 1994-99 programmes, a case study was carried out to assess the extent to which equal opportunities were being taken into account. On the basis of answers to questionnaires sent to all the entities promoting ERDF projects, the evaluator identified the projects most directly related to the application of equal opportunities. Several equal opportunities criteria defined by the evaluator were used to perform an in-depth analysis identifying projects with the best practices in this domain. In total, eight projects with the following potential effects were chosen: recognition of Transversal evaluation of impacts on intervention priorities equality, solidarity between women and self-esteem building, entry by women into traditionally male-dominated sectors, response to women's specific needs, promotion of dynamism among women, and reconciliation of their working and family lives. The in-depth analysis of these eight cases helped to identify the characteris­ tics of projects which promote equal opportunities, such as the adaptation of the form and content of training actions for women, the articulation of measures to ensure their coherence in terms of satisfying equal opportuni­ ties, the promotion of women's participation in project management, the involvement of women's associations in partnerships, the guarantee of women's participation in project monitoring, and so on (see Box 29). Box 29 - Standard form for assessing the equal opportunities dimension in the framework of ex ante evaluation Name of the entity Title of the project Territory Objective (1,2,. ) The project Objective Application of equal opportunities objectives (typology of effects) Typology of project promoters (% women) Beneficiary public (% women) Description of innovative actions and good practice Justification of equal opportunities strategies System of organisation and facilities developed Project programming framework in the ERDF Funds Programme Objective covered Axes, measures et actions Public and private co-financing Demonstrative value of actions Diffusion of results at local level Diffusion of results at regional level Transfer of results or methodology to other projects Multiplier effect Suggestions for the ap plication of "gender Mainstreaming": • Analysis of implementation of Finnish programmes. A thematic evaluation of equal opportunities was based on a matrix to assist commissioners and / or evaluators in their strategic evaluation choice and in the formulation of evaluative questions. This approach emphasises structure, process, results and impacts on women and men's lives and on the socio-economic environment, and provides evaluation criteria which take into account the principle of equal opportunities. The criteria concern Evaluating impacts on equal opportunities the different roles of men and women, women's access to financial aid provided by the Structural Funds, and the real needs of women in a specific context (see Box 30). Box 30 - Matrix for programme assessment Objectives of the assessment Roles Access to funds Needs Equal opportunities criteria Structure Process 1. programming 2. Implementation 3. monitoring 4. evaluation Results Impact The matrix facilitated the selection of evaluative questions for which the eva­ luation had to provide a credible answer. For example, "what are the roles of women and men in the different phases of design and implementation of programmes?" or "to what extent does the raison d'être of the programme and its objectives correspond to the identified needs and priorities of men and women in the region?". The evaluation succeeded in showing that allo­ wances granted in the framework of the ERDF programmes did not corres­ pond to Finnish women's real needs. • An initiative for the evaluation of the application of equal opportuni­ ties in the framework of objective 2 programmes (Sweden) Within the framework of IQ-NET (Networking to improve the quality of objective 2 programmes), a thematic study of the incorporation of equal opportunities into these programmes was performed. The aim was to facili­ tate the exchange of information and experiences between the regions, with regard to the design, implementation, monitoring and evaluation of the equal opportunities dimension of objective 2 programmes. The thirteen objective 2 regions belonging to this network are all characteri­ sed by a particularly high unemployment rate among men. The priority of the intervention is thus to promote the creation of jobs. The main object of the study was, however, to examine the contribution of objective 2 programmes to the promotion of equal opportunities. Within the objective 2 framework, some regions incorporate the promotion of equal opportunities into the Transversal evaluation of impacts on intervention priorities implementation of projects and then into their evaluation. Several case studies were performed to answer the following evaluative question: "What are the employment effects of the objective 2 programme, bearing in mind the transversal priority given to equal opportunities by the Structural Funds policy?". The results of the case study on the Swedish MASS project are worth pre­ senting. The project, inaugurated in January 1995 in the region of Ånger­ manlandskusten, was aimed at providing administrative assistance to several SMEs and at helping them improve their management. Jobless women were the main beneficiaries of the project. A pilot study was first conducted to define the problems confronting SMEs. It appeared that they encountered problems recruiting skilled workers. The second step related the needs of these SMEs to job applications. 14 persons were subsequently recruited, of whom seven were women pre­ viously employed in the public sector. Five of them were employed on a permanent basis. The project, above all, helped to reduce female unemployment and to improve the competence of the SMEs involved. It also contributed to reducing the workload of managers in these firms and thus to increase the time they spent with their families. • Evaluation of the participation of women in actions co-financed by the ESF This evaluation, conducted in 1991, was the first thematic study on equal opportunities and on the role of women in the Structural Funds. The origina­ lity of the study lay in the incorporation of the equal opportunities dimension in all the processes and methods of programming, implementation, mana­ gement and evaluation of ESF interventions. The construction of the methodological framework was based on objectives defined by the Commission: • qualitative and quantitative analysis of the role of women in programming documents; • analysis of the extent to which of equal opportunities were being taken into account in the processes and methods of programming, implementation, monitoring and evaluation; • qualitative and quantitative analysis of the role of women in the actions implemented; Evaluating impacts on equal opportunities • analysis of the coherence and complementarity between actions financed by the ESF and programmes and actions set up nationally and regionally to promote equal opportunities, • analysis of the coherence, synergy and complementarity between the NOW programme and actions financed within the framework of objectives 1,3 and 4. For the quantitative analysis, multiple and diverse sources of information were used, in addition to the CSFs and OPs, e. g. annual reports established either by the national and / or regional authorities, or by the training agencies; or even applications for support. For the qualitative analysis, interviews and / or questionnaires addressed to the actors involved in the preparation and implementation of the CSFs and OPs were used, e. g. ESF Missions; national, regional and local authorities responsible for employment and occupational training; training agencies and private enterprise; and promoters and managers of actions co-financed by the ESF. National and / or regional authorities responsible for policies in favour of women, as well as correspondents in each Member State of the Networks dependent on the Unit for equal opportunities (DGV), were also interviewed. National experts along with national and / or regional managers responsible for the preparation and implementation of NOW in the Member States were also contacted. The indicators used for the quantitative analysis concerned: • resources allocated to women, through mixed and specific actions, as compared to all available resources; • number of women beneficiaries of mixed and specific actions, compared to the total number of beneficiaries; • percentage of women beneficiaries by type of action, category of benefi­ ciary, level of qualification and sector of activity; • proportion of actions and national programmes in favour of training and employment of women, benefiting from ESF co-funding. The methodology made provision for the following four typologies: types of measures / actions, categories of beneficiaries, sectors of activity, and quali­ fications. It included a series of key questions, grouped together by theme. It also provided for the constitution of a sample of mixed actions for each type of measure, spread between actions with a high rate and those with a low rate of female participation, for the purpose of carrying out case studies. Transversal evaluation of impacts on intervention priorities ■ S u m m a ry The promotion of equal opportunities between men and women is a priority for which very little evaluation experience exists, this is particularly the case for ex post evaluation. The incorporation of equal opportunities into Structu­ ral Fund evaluations is therefore an important challenge. Actions focusing on the development of human resources are those most likely to produce a direct impact on the situation of women. This type of impact is easier to evaluate than indirect impacts on the promotion of equal opportunities. Evaluation can play an essential role in promoting the strategy of gender mainstreaming by ensuring that the principle of equal opportunities is syste­ matically taken into account in structural policies. ■ For m o re information see. Àlvaro­Page, M. (1996), Los usos del tiempo como indicadores de la discri­ minación entre géneros, Madrid: Ministerio de Asuntos Sociales; Instituto de la Mujer, N°46. Bamberger, M. (1996) "Understanding the Impact of Development Projects on Women: The Tunisia Institutional Development Fund Project" in Evalua­ tion for the 24st Century: New Topics for Evaluation. C. E. (1994), Les femmes et les hommes dans l'Union européenne: portrait statistique, Luxembourg: Office des publications officielles des Communau­ tés européennes. C. E. (D. G. V) (1998), 100 mots pour l'égalité, Glossaire de termes sur l'égalité entre les hommes et les femmes, Luxembourg: Office des publications offi­ cielles des Communautés européennes. Rubery, J. ; Fagan, C. and Maier, F. "Occupational Segregation, Discrimina­ tion and Equal Opportunity", pp431­461 in Schmid, G. ; O'Reilly, J. ; and Schömann K. (1996) International Handbook of Labour Market Policy and Evaluation, Cambridge: EE, University Press. Thematic evaluations of domains of intervention In addition to interventions targeted at promoting political priorities, interven tions within the framework of the Structural Funds take place directly in a wide range of domains. These domains are largely situated upstream of prio rities and may consequently have an impact on them. For example, interven tions in the domains of transport or tourism have possible environmental impact, interventions in favour of RTD are designed to have an impact on competitiveness of industry, in particular SMEs. The distinction between domains of intervention and political priorities has repercussions on the organisation of thematic evaluations. While the evalua tion of priorities can take place at any level above that of measure, thematic evaluations of domains will be transnational in nature, taking place across programmes. The selection of domains has been made on the basis of recent transnational evaluation experience acquired within the framework of the Structural Funds. The first chapter concerns interventions in the domain of RTD. There is a considerable amount of evaluation experience in this domain, but evaluation practice is undergoing a period of adaptation to match the evolving policy context. This structure of this chapter is similar to that of the chapters on the four political priorities in Part II. The second chapter is, on the other hand, a rapid survey of evaluation practice and issues relative to the domains of transport, telecommunications and tourism. Evaluating impacts arising from interventions in the domain of Research and Technological Development RTD ■ W hy evaluate the effects of interventions on RTD? For regions depending on agriculture and more traditional labour intensive industries, the opening up of national markets has meant increased compe­ tition from lower cost regions both within and outside the European Union. This has consequently led to a reduction of low skilled employment opportu­ nities and has exacerbated problems associated with economic develop­ ment and industrial restructuring. The decline of traditional sectors has been accompanied by an increasing reliance on new, rapidly expanding industries to generate economic growth. Because firms in these industries tend to be small or medium­sized, with generally limited resources for RTD investment, their research effort often has to be undertaken in co­operation with other firms or with research insti­ tutes to obtain optimal results. However, this type of collaborative arrange­ ment tends to occur most often in specific circumstances of which geogra­ phical proximity, the existence of non­pecuniary relations and training­ research­industry linkages are important factors. Particular examples of these phenomena that have inspired recent public policy thinking in the domain of RTD are the Italian "Districts" and the close links between training systems and industry found in Germany. As industrial restructuring has progressed, it has become increasingly evident that a process of technological polarisation is underway. The more technolo­ gically advantaged regions benefit from prior technological capacity, continued high levels of RTD expenditure, a well developed innovation culture in general, and the ability to further develop their innovatory capacity, whilst the more technologically disadvantaged regions fall further and further behind. This divergence of RTD capacity is reflected in the persistence of regional disparities in terms of competitiveness, trade, unemployment and growth, both within individual countries and across the EU as a whole. New insights into the process of innovation have meant that the linear "Science­Technology­Production" type model has given way to the concep­ tualisation of innovation as a dynamic, interactive and non­linear process. In addition, the understanding of the territorial dimension of RTD, through the framework of regional and national systems of innovation, underpins the Transversal evaluation of impacts on intervention priorities rationale of interventions aimed at economic and social cohesion. As a result of these advances, more emphasis is being placed on the creation of endo­ genous technological capacity and on the diffusion of innovations to SMEs through increased co-operation between universities, research institutes and enterprises, as well as support and promotion of industrial RTD. This change of orientation reflects the fact that earlier interventions have had generally disappointing results in regions with low RTD potential. Central to this change of emphasis is that, with the demise of the linear model of innovation, policies focusing on the "technology push" approach are inadequate. It is now realised that technology supply and demand have to be matched through an interactive process involving both producers and users of new technologies. Moreover, the development of a region's innova­ tory capacity is intrinsically linked to the tailoring of interventions to suit the character and to anticipate the evolution of its productive structure. Where SMEs predominate, one of the major requirements in RTD intervention is simply to help them appreciate the importance of innovation for competitive­ ness in an open economy. The present generation of interventions are therefore largely composed of "mission oriented", bottom-up measures aimed at the wider business and commercial needs of SMEs, their demand for outside technology and integration of it into their production processes, their own capacity in product innovation and marketing, the financing, orga­ nisation and management of innovatory activities, and the creation of new high-tech firms. • The Community context The importance of RTD for the competitiveness of a region and the need to develop RTD capacity is recognised within the Structural Fund framework. As a result, Structural Fund support for RTD has risen dramatically during a period in which national governments have generally been cutting back on science and technology (S&T) expenditure. RTD interventions are financed under Structural Funds, with the ERDF being the major source. RTD support figures prominently in Community Initiatives (INTERREG II, PME, ADAPT. ) and is also the subject of innovatory actions: Regional Innovation Strategies (RIS) and Regional Technology Transfer (RTT). In addition, leading areas of scientific and technological research are the object of interventions associated with Community Support Framework programmes. Evaluating impacts in the domain of RTD Within the global aim of the EU's RTD policy, which is to strengthen European competitiveness through the development of scientific and tech­ nological excellence, Structural Fund interventions are targeted at counte­ racting the concentration of RTD capacity in certain countries, mainly in the North of the EU (transnational cohesion), and in certain regions, mainly in and around countries' capitals and a few dynamic regional poles (national cohesion). Structural Fund resources have therefore been directed to more peripheral regions with the object of developing their S&T base, particularly through the establishment of sustainable links between the public research bodies and regional industry. Over the long term, these interventions should increase the innovatory capacity of assisted regions either by reorienting their technologi­ cal capacity towards the faster growing, high technology sectors or by creating indigenous technological capacity and therefore reducing the "technology gap" with developed regions. The longer-term impact of this process will be increased regional specialisation in high value-added products and services, increased growth, and enhanced economic and social cohesion. Recent experience of interventions in the domain of RTD, for the 1989-1993 programming period, has shown that a number of problem areas in RTD funding exist in the regions. Firstly, the relationship between increased RTD funding, increased innovation and economic development is neither automatic nor direct. Attention needs to be paid not only to technology supply, usually generated upstream from industry in public institutions, but also to the demand for technology, fashioned by the needs of a region's industrial actors and their ability to match available technologies with com­ mercial opportunities. However, funding during this period tended to be concentrated in public research and did not consequently address the practical RTD needs of local firms, particularly those of SMEs. In the least developed regions there are major problems of absorption of available funds due to the fact that suitable projects are often difficult to identify or authorities have little experience with or understanding of RTD issues. In certain regions this problem has aggravated the concentration of funds in public RTD upstream, with the consequent mismatch between public technology supply and industrial technology demand (see Box 31). Transversal evaluation of impacts on intervention priorities Box 31 ­ Evaluation of the impacts of CSFs for Research and Technology in Greece, Ireland and Portugal for the 1989 ­ 1993 programming period The objective of this evaluation was to assess the impact of interventions, make comparisons between the 3 countries studied and make recom­ mendations which would help improve both the efficiency and the effecti­ veness of CSF support for RTD in these countries. The study found that the national authorities in the three countries had largely used the oppor­ tunities provided by the funds to considerably enhance public infrastruc­ ture for RTD. It also reported that all three countries had had problems in the areas of industrial participation, technology transfer, overemphasis on technology supply, concentration of RTD around capital cities and human resource shortages. Problems of operational costs, achieving self­ financing targets, and management and evaluation also occurred. Conse­ quently, during the second programming period (1994­1999) more emphasis is being placed on better use of existing RTD capacity and less on building additional RTD capacity. The development of European RTD capacity in general, and in relation to cohesion in particular, represents an important strategic objective of the EU. This is reflected by the fact that interventions aiming to develop the techno­ logical potential of the regions amount to 7,600 million euros, approximately 7% of all Community spending/ The evaluation of RTD interventions therefore has a central role to play in assessing the extent to which they are capable of achieving (or are achieving or have achieved) stated policy objec­ tives. The results and conclusions of these evaluations can then be used to inform the decision making process and consequently enhance the effecti­ veness of policies and programmes. A good example of this process is the evaluation of the impacts of CSF on research and technology in Greece, Ireland and Portugal for the 1989­1993 programming period (see Box 31). ■ Defining w h at needs to be evaluated When evaluating Structural Fund interventions in the domain of RTD, it is important to bear in mind that the undertaking of RTD of sufficient quantity and quality is not an aim in itself. Nor is it a factor which when introduced into policy, leads to convergence towards a desirable goal. It is a prerequisite of sustained economic development and socio­economic cohesion. 4. "Les Fonds structurels en 1996, huitième rapport annuel". This figure includes interven­ tions in the domains of R&D and innovation. Evaluating impacts in the domain of RTD Whilst it is relatively straightforward to evaluate outputs and results of RTD interventions, it is more difficult to estimate the impacts produced. Specific impacts are often of an intangible nature (e. g. qualitative changes in human capital) and impacts at the socio-economic level only occur over an extended time period, making impacts difficult to attribute to interventions. The techno-industrial structure of a region is fashioned over time by a multi­ plicity of factors. As a result, the needs of regions and therefore the extent and type of RTD intervention vary greatly from one region to another. For instance, in lagging regions and to a lesser degree in those with low popula­ tion densities, the supply of facilities and equipment, the number of resear­ chers, the application of research results and the transfer of technology to local industry are problems that have to be addressed if endogenous scienti­ fic and technical capacity is to be created. It is, however, also pertinent to question to what extent interventions in regions with little history of research and technology should be undertaken at all. As for declining industrial regions, interventions are generally oriented towards support for investment in specific high-tech sectors (e. g. biotechnology, microelectronics, etc. ), investment in key locations and the creation of investment projects to employ highly trained personnel. • Some key issues in RTD evaluations There are a number of key issues related to structural support for RTD that can be used to orient the evaluation of RTD interventions: • Effectiveness in promoting RTD capacity and capabilities: What consti­ tutes RTD capacity and capabilities? Do interventions promote the intended results and impacts in terms of RTD capacity and capability? It should be noted that because the benefits of a single research project are not usually captured by or do not generally lead to specific innovations, particularly where collaborative research is concerned, the results of cost-effectiveness type analyses may be of limited worth unless RTD activities are appraised at a portfolio level. • Strength of linkages generated by interventions within the local economy: Whilst measures of activity (e. g. the number of SMEs involved in collabora­ tive RTD projects) may be high, the intensity of involvement may differ from firm to firm. Focusing on the financial participation of the recipient firms or some sort of qualitative assessment of collaborative ventures can help to give an indication of the strength of these linkages. • Skills shortages and improved qualification levels: In regions undergoing Transversal evaluation of impacts on intervention priorities industrial restructuring, high unemployment coexists with skills shortages in some of the more technologically advanced sectors. However, an important question is the extent to which benefits of increased educational and training opportunities, especially at post-graduate level, are lost through the outward migration of addressees of supported training to other regions? • When improved economic performance is primarily a consequence of process innovations leading to reduced labour costs, initial effects on employment are likely to be a reduction in jobs. This may often be the case in regions where former labour intensive industries in declining industrial regions are attempting to remain competitive through capital investment and labour substitution. The relationship between innovation and employment creation or maintenance therefore warrants particular attention. • Interactions between interventions and national RTD policy, and the regional distribution of RTD activities: The geographical distribution of public research capacity is usually strongly influenced by national strategic consi­ derations. Consequently, in more centralised states this leads to research facilities tending to be concentrated around the capital city. Conversely, in more decentralised states research capacity is usually found to be less cen­ tralised, though it may be concentrated around regional capitals. Thus, national RTD policies and Structural Fund interventions can display varying degrees of positive or negative synergies. • Synergy and integration with other EU interventions: Whilst Support Framework programmes have, until recently, been concerned with more upstream "pre-competitive" research projects, and Structural Funds with enhanced research capacity and technology transfer, the reality of the situation is that the boundary between these approaches is, in fact, increa­ singly blurred. This overlap suggests that there are possibilities of synergies and / or dangers of double employment between these two policy instru­ ments. Furthermore, research projects selected under the Framework programme are chosen on the basis of excellence and without regard to cohesion and structural objectives. This generates negative synergies between these instruments when research capacity, as is generally the case, is concentrated in the more dynamic and wealthier regions. • Future policy orientations: are structural interventions capable of overco­ ming the strong tendency towards geographical concentration of RTD, indeed, is this objective relevant? Or, on the contrary, should interventions aim to concentrate RTD capacity in a relatively small number of regions so as to achieve critical mass? Should Support Framework programmes be tailored to take account of the concerns of structural adjustment and Evaluating impacts in the domain of RTD cohesion? How can overlaps between the two policy instruments be reduced and how can synergies be encouraged? How can problems of absorption and additionality be overcome? The choice of level of study of RTD activities is largely determined by the need to take into account the externalities created by research projects. Because scientific and technical knowledge are to a certain extent public goods, the estimation of effects at the project level are likely to seriously underestimate the benefits of the project accruing to society as a whole. It is for this reason that RTD is being increasingly studied either at a meso- economic or macro-economic level. Since Structural Fund interventions are geographically delimited, a macro approach seems to be the most relevant level of analysis. Analysis at this level also allows interventions and projects to be examined in the larger context of the regional or national innovation system of which they are an integral part. This does not exclude the possibi­ lity of integrating a meso-level analysis into an evaluation where a sector or filière is of particular importance in a region. There are four main channels through which Structural Funds intervene to reshape a region's RTD profile: • Support for S&T infrastructures: universities, associated research centres and public laboratories, and structures to promote the diffusion of results of research projects. Activities in this area are aimed at enhancing scientific and technological potential; • Direct support for industry: subsidies and soft loans to support R&D and technology transfer activities, and the creation of high-tech firms. Activities in this area aim at enhancing technological potential, increasing links between research and innovation and improving the financial framework for performing RTD; • Indirect support for industry: services such as technology audits and expert advice, etc. , strengthening of innovation and technology centres, and efforts to increase participation in other EU RTD initiatives (Support Framework, Community Initiatives, etc. ). These activities are intended to enhance technological potential and to support innovation through improved links with research; • Training: training for researchers and technology managers with an overall aim of promoting an innovation culture. The suggested framework for evaluating RTD interventions presented in Box 32 is based on two broad categories of activities: the strengthening of S&T Transversal evaluation of impacts on intervention priorities potential and support for innovation. These categories group together a number of themes and sub­themes, some of which may warrant a more focused analysis depending on the RTD profile of the region under examina­ tion and the type of intervention. Box 32 ­ Framework for the evaluation of Structural Fund interventions in the domain of RTD Structural Fund interventions in the domain of RTD Strengthening ( 1 1 ■ I Enhancer of scientific potential Construction of university research centres and Grants for equipment Grants for projects Etc. 1 1 1 1 1 1 1 of an 111 links between I II of financial II "innovation II research | || framework for 11 E ES | " ' ■ ­ " ■· * I ï 1 Technology transfer 1 1 Seed and venture capital Construction of technology centres, science parks, support for SME research capacity Grants for equipping technology centres, measurement and testing equipment Training Researcher mobility Stimulation of activity Diffusion of results Grant funding of development projects Awareness raising activities Etc Etc. Etc. Promotion of Etc. Linkages between firms, universities. centres of research • Clarifying the objectives of RTD interventions In the case of RTD interventions, an objectives tree can be used to link up the outputs and results of RTD interventions to the specific objectives envisaged within the framework of the Structural Funds (see Box 33). Evaluating impacts in the domain of RTD Box 33 - Example of a thematic objectives tree for RTD interventions G l o b al o b j e c t i v es E x a m p l es of s p e c i f ic o b j e c t i v es Increased technology transfer between institutions and enterprises • Increased number of collaborative projects between SMEs and research institutes • Increased number of post-graduate in-firm placements Increased Innovation by enterprises, particularly SMEs • Increased number of inter-firm RTD collaborations Involving SMEs • Increased number of SMEs performing RTD Improved RTD Infrastructure • Increased number of S&T parks • Improved university research facilities • Increased number of new university research facilities Increased availability of suitably qualified researchers and technicians • Increased number of places on science and technology oriented university courses • Increased number of research grants available Links between intermediate objectives Reduced inter· regional and infra-regional disparities in RTD capacity Increased economic and social cohesion While specific objectives are strictly subordinate to the intermediate objec­ tives in the tree, the latter are not independent of each other. For example, attaining the objective of improved RTD infrastructure is likely, by providing facilities for firms wishing to perform RTD, to contribute to innovation by SMEs. While giving an overview of the structure of the objectives of an intervention, the objectives tree also provides an indication of what an impact typology for RTD-oriented Structural Fund interventions might look like. Using the inter­ mediate objectives of an intervention as a starting point, a typology of RTD impacts might look like this: • Technology transfer; • Industrial RTD; • RTD infrastructure; • Human resources. Transversal evaluation of impacts on intervention priorities • Net effects of RTD interventions The net effects of structural interventions in the domain of RTD are difficult to evaluate. Impacts, especially at the socio-economic level, often occur a long time after RTD investments have taken place, which makes their attribution to individual interventions difficult to ascertain. RTD impacts are heteroge­ neous and must therefore be estimated using an array of indicators: sales of innovatory products, processes and services; R&D employment created; technologies acquired; networks created; number of PhDs obtained, etc. They cannot therefore be summarised in the form of a single indicator. Generally speaking, the estimation of net effects emphasises the assess­ ment of deadweight (i. e. the change which would have occurred to the activity under examination in a policy-off situation). However, it is important to note that although public funding dominates upstream research, at the industry level the private versus public funding balance varies between sectors and the importance of deadweight will, as a consequence, vary accordingly. It is for these reasons that the estimation of net effects of RTD interventions is generally undertaken at the results level, by focusing on changes in RTD expenditure resulting from an intervention in the domain. Example: From gross to net effects in the support for innovation in SMEs Assuming that support for a venture capital fund to finance research projects performed by SMEs has led to 435 projects being launched at a cost of 600 million euro. It is however found that 120 projects costing 150 million euro would have gone ahead without support (deadweight). The net effect of the intervention would therefore amount to 450 million euro. Another factor which has to be taken into account is the leverage effect (i. e. the increase in privately financed RTD which results directly from Structural Fund expenditure in RTD projects). Leverage along with substitution is often explicitly sought by policy makers. The latter results from interventions aiming to persuade high-tech enterprises and research institutes from outside the target region to relocate within the region in order to increase its RTD capacity. In the case where diversification of the regional economy is a prime consideration, substitution of innovative activities for activities those in declining low-tech sectors will also be an explicit objective. Evaluating impacts in the domain of RTD ■ Examples of i m p a c ts by area of intervention A number of different examples of RTD projects are presented below, cor­ responding with the three broad areas of intervention of the Structural Funds. * Basic infrastructure Interventions in the area of basic infrastructure usually have an indirect effect on RTD. For example, improvements in telecommunications networks may increase the scope for using telematics for long distance RTD collaboration. Infrastructure co­financed by Structural Fund interventions, having a more direct effect on RTD, includes facilities for university research, science and technology parks, etc. Example: The construction of a science and technology park in Northern Italy The Tecnoparco del Lago Maggiore in Piedmont was conceived to help entrepreneurs interested in technology and innovation to turn their ideas into successful businesses. The park, co­funded by the ERDF and the Piedmont region, provides rented facilities for research projects. Tenants of the Park also benefit from back­up services (business counselling, environmental safety and certification, computer­aided design, partner and finance search and the maintenance of meeting and training facilities) and support for innovative technology projects. The long­term objective of this project is to attract research investment and initiatives from outside the region and to stimulate the region's economy, allowing traditional industries to be replaced by innovative SMEs. Outputs O Construction of buildings and provision of equipment Firms setting up in the park Effects New firms becoming established Jobs directly created by the firms located In the park Transversal evaluation of impacts on intervention priorities • Productive environment Interventions in the area of the productive environment cover a large spectrum of possible measures ranging from direct support for RTD invest­ ments to support for services aimed at assisting the technological activities of firms, particularly SMEs and NTBFs, and at creating networks involving both public and private institutions. Example: Promoting technology transfer in Aquitaine (France) The economy of Aquitaine is dominated by the presence of large enterprises in the armaments, space and aeronautics sectors. These sectors have, for a number of years, been undergoing large scale restructuring which has had dramatic results not only on their own activities, but also on the activities of local subcontractors in a diverse range of sectors. One of the central priorities established in the 1994-96 SPD for the region was to adapt the existing potential of the region's SMEs through a coherent policy for innovation and technology transfer. One of the interventions within the framework of this policy has been the setting up of CREATI Aquitaine, (Centre Régional d'Appui Technique et d'Innovation) which in 1997 brought together 10 large regional enterprises committed to developing a support network of technical services for local subcontractors as well as promoting technology transfer. The CREATI has also been involved in a number of projects launched in partnership between partici­ pants and SMEs. These projects have also benefited from co-operation with regional universities. Outputs > Effects O Advice sessions financed with experts in technology Issues SMEs entering into collaborative arrangements Production of new products, processes and services incorporating acquired technology Evaluating impacts in the domain of RTD • Human resources Interventions in favour of human resources cover such measures as financing university research grants in specific areas (e. g. technology transfer, etc. ), vocational training in high­tech subjects, encouraging links between public research and industry (e. g. placements for researchers in industry, etc. ). Example: Centros Tutelados (Spain) This project, set up by the Basque government in 1992, aims to develop a technologi­ cal network, support growth and create j o bs in local industries. The Centres support the development of local enterprises by undertaking their o wn research, by monitoring of scientific p r o g r e ss a c r o ss the w o r l d, a nd by their t r a i n i ng p r o g r a m m es w h i ch represent one of the major aspects of the centres' activities. Highly qualified technical and research staff undergo a t wo year training period in the Centres before going to work in regional SMEs to ensure that the knowledge they have gained is diffused into local industries. The Centres also organise training courses for local enterprises. Up to 1997 over 1,400 trainees in nearly 900 enterprises had benefited from this service. Outputs > Organisation of external training courses Financing of places for technicians and researchers within the Centres Effects > Increased qualifications and skills of employees of client firms Increased overall skills level within participant firms Placing of researchers in regional companies Companies producing new or improved products, processes or services ■ Defining and selecting indicators The evaluation of RTD interventions requires the construction of a system of indicators using both standard RTD indicators and ad hoc indicators that take into account the particularities of the specific evaluation. The monito­ ring of interventions should generally provide data for resource, output and result indicators, whilst the gathering of information on impacts requires the use of active data collection techniques. Resource indicators such as GERD, BERD and GOVERD5 give information on the structure of RTD financing and the consequent distribution of activity between public research and industrial RTD. When related to population or GDP, comparisons can be made between regions and countries. 5. GERD ­ Gross Domestic Expenditure on Research and Development; BERD ­ Business Expenditure on Research and Development; GOVERD ­ Government Expenditure on Research and Development. Transversal evaluation of impacts on intervention priorities Output indicators focus on the activities of operators (contacts with addres­ sees, applications processed, etc. ), the number of hours of consultancy and training sessions, number of supported research projects, networks and colla­ borative ventures financed, and the creation of infrastructure such as science parks, research facilities, etc. (measured in square metres or hectares). Result indicators, in the case of RDT infrastructure could provide information on the quality / cost ratio of, for example, establishing a firm in a science and technology park, or the attractiveness of the park and its environs to resear­ chers. As for technology services, results indicators largely focus on measure addressees' satisfaction. Indicators at this level should also measure the amount of private money invested in the supported projects, the leverage effect of an intervention. Another useful indicator is the proportion of addres­ sees that undertaking RTD for the first time with the support of the programme. Results indicators should also provide information on networking between firms and institutes when this is the objective of an intervention. Impact indicators focus on wider and more long-term technological and socio-economic impacts. Because the developmental effects of RTD occur over a long period and some specific impacts are difficult to measure (i. e. the knowledge and know-how acquired by individuals), impact indicators are generally limited to those informing on specific socio-economic impacts of a more tangible nature, for example, the number of new products and processes marketed, their sales and the sales of related patents and licences, increased value added and direct employment effects (researchers and technicians measured in FTEs). Because public and private research are increasingly intertwined - indeed, collaboration between public research centres and SMEs and the joint development of innovations is one of the objectives of interventions - many institutes are themselves starting to behave like private firms: patenting innovations, selling licences and marketing products, etc. Indicators for university and research centre activi­ ties within the framework of Structural Fund interventions should therefore focus on the more industrial and commercial manifestations of public sector research. By way of an example, a system of indicators that could be used to track the production of impacts arising from a project providing financial and informa­ tional support for technology acquisition by SMEs is presented in Box 34. Evaluating impacts in the domain of RTD Box 34 - Example of a system of indicators for an intervention providing financial and informational support for technology acquisition by SMEs Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Activity of the operators Number of contacts between operators andfimns looking for support for technology acquisition Number of advice sessions organised with experts in technology acquisition Number of addressees Number of SMEs receiving support Volume of technical services provided Hours of consultancy and training services dispensed Result indicators Satisfaction % of addressees satisfied or very satisfied with the service Leverage effect Investment in acquisition oriented RTD by firms benefiting from financial support Ratio of BERD to GERD Impact indicators Internal technology transfer External technology transfer Technology acquisition Economic impact Number of collaborative arrangements between research institutions and assisted enterprises after one year Number of assisted firms purchasing patents, licences or entering Into R&D collaboration with foreign firms after one year Number of new products or processes incorporating the acquired technology marketed by assisted firms after one year % of managers of recipient firms who declare that the Intervention has relaxed constraints in terms of access to new technologies % of regional managers who declare that access to new technologies is a major constraint Value added Employment Value added generated after one/three years in euro / year Value added generated in the region by SMEs Net employment created after one/three years (of which women) Total number of jobs in SMEs (FTEs of which held by women) Source: Volume 2 of the MEANS Collection Transversal evaluation of impacts on intervention priorities Whilst these indicators concentrate on the quantification of outputs, results and impacts, it is also useful to have qualitative information about the effects of interventions. For example, data on the number of collaborative projects between research institutes and firms, or the number of innovations arising from them can be enriched by qualitative information about the venture. But, because this sort of collaboration is intended to improve the competitive position of the participating enterprises, for reasons of commercial confi­ dentiality standard peer review or expert panel procedures used for the eva­ luation of public research institutes are not particularly suited to this task. However, partners could be asked to assess the quality of their venture. A possible scoring system for measuring the quality of collaborative ventures is presented in Box 35. Box 35 - Example of a score card for a joint research venture Assessment of the: • Strategic relevance of the joint research activity to your commercial activities 0 12 3 4 • Quality of the research performed within the project 0 12 3 4 • Quality of the contribution made by your partner(s) 0 12 3 4 • Extent of learning engendered in your organisation by the partnership 0 12 3 4 Legend: 0 - Very Poor; 1 - Poor; 2 - Low; 3 - Good; 4 - Excellent Evaluating impacts in the domain of RTD ■ Suitable sources of information When evaluating RTD interventions within the framework of the Structural Funds, it is important to remember that the tools used must be suited to examining complex phenomena in detail. One must also bear in mind that the availability of monitoring data at the regional level is sometimes rather patchy, so that in some regions primary data collection will be an expensive and time­consuming part of an evaluation. Whilst the evaluation of RTD interventions is therefore likely to be heavily dependant upon primary data, sources do exist which provide relevant sta­ tistics at the firm level. Innovation surveys focus either directly on innovations or on firms and their innovatory activities. In the first instance, survey data on innovations deemed technologically or economically significant are collected. The second type has the advantage of taking into account both innovatory and non­innovatory firms as well as the users of innovations, allowing factors which restrict innovation and the existence and intensity of user­producer linkages to be examined. Data on patents is also available at the firm level, giving the technological classification of the innovation. Unlike data resulting from innovation surveys, patent data is available over a long period. Patents registered at the European Patents Office are available on the EPAT database, which has been in operation since 1978. Data are generally available at the national level (e. g. l'Observatoire des Sciences et les Techniques produces an annual report on S&T indicators for France) and statistics on national R&D expenditure are provided annually by the OECD. Relatively up­to­date European RTD statistics suitable for constructing context indicators are available from Eurostat in an annual publication which covers data on R&D personnel, R&D expenditure and patent applications broken down by region as well as government budget R&D appropriations. This source also provides "nearcasts" that provide estimates of current sta­ tistics that are being compiled. ■ Appropriate tools for evaluating structural interventions in the domain of RTD While a detailed treatment of evaluation tools is provided in Volume 3 of the MEANS Collection, a number of tools particularly suited to evaluating impacts on RTD are presented below. Transversal evaluation of impacts on intervention priorities While a great deal of experience has been accumulated in this domain, and standard tools for this type of intervention have a long track record, the increased importance of interventions towards generating "opportunities" for innovation through the creation of better conditions for learning, partner­ ships and networks, has led to new challenges in the area of RTD evaluation. First, these effect arising from RTD interventions are difficult to realistically reduce to simple monetary variables, which means that tools such as cost- benefit analysis and return on investment type analyses are of limited use. This latter technique is restricted to analysing private returns arising from RTD expenditure, it therefore significantly underestimates total returns, since a large part of these are social returns generated through RTD spillovers and externalities (i. e. effects, either positive or negative, that to some extent take place through non-market mechanisms and are therefore not fully accounted for within the price system). Secondly, because of the complexity of the chain of causality between RTD activities and their socio-economic effects, increased emphasis has to be placed on the use of tools allowing in-depth qualitative study. Thirdly, the recognition of the effects of RTD spillovers on the performance of firms requires that evaluations simultaneously take a wider approach than that afforded by project-by-project examinations of interventions. Finally, the evolution in the nature of RTD interventions, the increased emphasis on eva­ luating interventions in the area of industrial RTD and the need to take fully into account the spatial objectives of structural interventions, mean that eva­ luation techniques in this domain are less able to rely on classic RTD evalua­ tion tools such as expert panels and bibliometric indicators in which a great deal of experience has been accumulated. In terms of the alternative approach required, concrete evaluation expe­ rience remains rather limited. The tools presented below are those which have to a limited extent been used in evaluations in this domain or seem par­ ticularly suited to examining impacts of RTD oriented interventions. By using questionnaire surveys that are carefully tailored to the characteris­ tics of the population concerned, relatively simple, and clearly structured and designed to cover of the RTD themes depending on their importance to the region or regions studied, evaluators can build up a picture of a region's innovation system in terms of the type and extent of interventions, how they are implemented, the nature and extent of industrial RTD, the effect of inter­ ventions on regional RTD policies and the organisation of RTD activities. Evaluating impacts in the domain of RTD Whilst informing on resource use, issues of organisational and management quality, the effectiveness of instruments and mechanisms, surveys should largely focus on providing quantitative data on the outputs and results of interventions. Questionnaires may also have to be designed to reflect the cover of programmes (i. e. regional or multi-regional). Experience from the EUREKA Initiative has shown that questionnaires alone are not able to capture the richness and complexity of socio-economic effects arising from RTD interventions. Furthermore, experience from Norway has shown that such impacts as extra sales, productivity gains, net­ working benefits and increased skill-levels may take over ten years to become evident. Questionnaires therefore need to be supplemented by face-to-face interviews with key figures in the domain of RTD, focusing both on actual and expected impacts. The interviewees will be drawn from programme actors: regional and local programme sponsors, universities, etc. More importantly, they will also include addressees such as research centres, SMEs and larger companies involved in programmes along with firms not directly involved (e. g. firms buying process innovations, rather than developing them themselves). Expert panels can be used for evaluating interventions targeted at public research. However, because secrecy is one of the ways firms maintain their innovative edge, they are of limited use when private or collaborative research activities need to be evaluated. On the other hand, expert panels are useful tools for undertaking "quick and dirty" evaluations, when these are required. A number of carefully selected case studies can be used to provide in-depth analysis and comparison of individual projects and measures. These might focus on projects of outstanding quality in the area of the promotion of RTD capability. This technique may be particularly useful in examining internal (e. g. the setting up of a RTD department) or external (e. g. linkages between firms or between firms and research centres) organisational changes engen­ dered by interventions. Case studies may also cover an entire region in order to better appreciate both the functioning of regional innovation systems and the long-term sustainability of a region's development. In this case they might look at competitiveness, economic development and employment, focusing on the demand side of RTD. Case studies should be designed with issues of extrapolation, aggregation and comparison in mind. Transversal evaluation of impacts on intervention priorities ■ S u m m a ry The importance of RTD, especially at the industry level, is recognised as an essential prerequisite of sustained economic development. This recognition has led to a large increase in Structural Funds expenditure in this domain, accompanied by the reorientation of interventions towards downstream technological development. When estimating the net impacts of interventions, in addition to identifying deadweight additionality of Structural Fund expenditure is an important factor that should be taken into account. The assessment of leverage effects also represents an important element in appreciating the impact of interven­ tions on RTD. The evaluation of RTD interventions poses particular problems because of the intangible nature of many specific impacts and the long­term nature of socio­economic impacts. Moreover, the increased emphasis on the role of private sector RTD and the need to evaluate at a higher level than individual projects, mean that evaluations are less able to rely on assessments of expert panels, the established tool of RTD evaluation. ■ For more information see. Archibugi D. and Pianta M. (1996), "Innovation surveys and patents as tech­ nology indicators: the state of the art" in: Innovation, Patents and Technolo­ gical Strategies. Paris: OECD. Commission européenne (1998), 2e Rapport européen sur les indicateurs scientifiques et technologiques 1998, Luxembourg: Office des publications officielles des Communautés européennes. European Commission (1999), Impact of Structural Funds (1994­1999) on Research Technology Development and Innovation (RTDI) in Objective 1 and 6 Regions, European Commission, DG XVI. Eurostat (1997), R&D Annual Statistics 1997. Luxembourg: Eurostat. Georghiou L. (1998), "Issues in the Evaluation of Innovation and Technology Policy", Evaluation, Vol 4(1), 37­51. Lundvall B. ­A. (éd. ) (1992), National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning. London: Pinter. OECD. (1994), The Measurement of Scientific and Technological Activities: Proposed Standard Practice for Surveys of Research and Experimental Development ­ Frascati Manual 1993. Paris: OECD. OECD. (1998), Policy Evaluation in Innovation and Technology: Towards Best Practices, OECD Proceedings, OECD: Paris. OST. (1996), Science & Technologie: Indicateurs 1996. Paris: Economica. Evaluating impacts arising from interventions in the domains of transport, telecommunications and tourism A small amount of experience in thematic transnational evaluations has been gained in the domains of transport, telecommunications and tourism. This experience can be characterised as both limited in terms of its transnational aspect and predominantly top­down in terms of evaluation design. The ambition of this chapter is to draw on this available experience and to com­ plement it with a number of points that should be considered when evalua­ ting interventions in these domains. ■ Transport and telecommunications • Structural interventions in the domain of transport Structural Fund interventions in the domain of transport are part of a larger Community transport policy which also includes EIB funding and the Trans European Network (TEN) initiative. In addition to these instruments focused mainly on the funding of transport infrastructure projects, a substantial body of legislation, geared towards the harmonisation and liberalisation of market conditions within particular transport sectors, now exists. Transport infrastructure projects are funded by means of grants and loans that are largely allocated on the basis of project­specific appraisal and regional development strategy. For the cohesion countries, from both a physical and financial point of view, the planning and financing of transport infrastructure are among the most important areas of EU policy. Transport infrastructure investments are intended to affect socio­economic cohesion in two major ways: • by eliminating bottlenecks and reducing travelling times, costs and constraints on industry (e. g. of transporting raw materials, semi­finished and finished products, etc. ) are reduced and economic productivity is improved (e. g. by allowing new organisational techniques such as just­in­time operations); • by increasing accessibility and mobility between regions, therefore reducing the effects of peripherality on business and individuals. This traditional argument is that improvements in transport and, in particular, transport infrastructure, foster economic development and reduce dispari­ ties between regions. This hypothesis is, however, questioned by some who argue that improved infrastructure may in some cases worsen disparities, at Transversal evaluation of impacts on intervention priorities least in the short-term, by opening up regions to imports originating from lower-cost sources. In addition to this possible undesirable effect, interven­ tions within the framework of the TEN initiative, which account for the majority of investments in this domain, are not aimed at achieving cohesion but at increasing the overall competitiveness of the EU as a whole. Interven­ tions of this kind may therefore have greater positive effects in non-cohesion countries in which the bulk of the investment in TEN projects occurs and which are more favoured in terms of accessibility by their geographical position. Nevertheless, the role of the Community in identifying, planning and co-financing the TEN is seen as an important factor in countering the national bias of most transport infrastructure projects. It also seems possible that the tendency of improved infrastructure to attract individuals and busi­ nesses to the more dynamic regional poles of activity at the expense of cohesion may be largely overestimated. In the medium to longer term it can be expected that there will be a degree of rebalancing of activity as the costs and disadvantages of large agglomerations (e. g. office rents, pollution, congestion, etc. ) start to outweigh the benefits. In addition to the upgrading and construction of infrastructure, there is a growing emphasis on more efficient use of existing infrastructure through high-technology solutions to problems of road congestion and transport related environmental problems (e. g. Advanced Transport Telematics tech­ nologies). The effect of this development on cohesion may prove to be less than was bargained for since this sort of solution seems more suited to developed rather than peripheral parts of the Community. The provision of transport infrastructure may also have negative effects arising from such problems as road congestion, due to the apparent capacity for demand for road space to outstrip supply. Negative effects on the quality of life may result from air and noise pollution generated by the use of the infrastructure. The evaluation of the effects of interventions in this domain will depend upon what is considered as the contribution of transport to the economy. That contribution largely translates into reduced costs and constraints for busi­ nesses and individuals. It also has a strong qualitative aspect, notably in terms of safety and environment. • Structural interventions in the domain of telecommunications The transition from an industrial to an information-based service economy holds a number of opportunities for the more peripheral regions, since the Evaluating impacts of interventions in the domains of transport, effects of geographical distance on business activity is consequently reduced. However, the possibility of benefiting from these opportunities, depends on these regions being well equipped in télécoms infrastructure and services. In light of advances in technology, which have meant that telecommunica­ tions and information technology are increasingly indistinguishable and that national telecommunications monopolies are no longer tenable, the role of the EU has been twofold: • First, it has put into place a common framework for a liberalised European télécoms sector within which the Member States regulate their telecommu­ nications industries; • Secondly, it has promoted collaborative télécoms RTD and supported the modernisation of télécoms infrastructure. The effects of liberalisation policies in the domain of telecommunications will lead to an overall improvement in service quality and efficiency. However, their short-term effects will probably accentuate problems of cohesion. The immediate effects will be a reduction in costs for heavy consumers of télécoms services in the more developed regions as the cross-subsidisation of services stops and tariffs are rebalanced to reflect real costs. For consumers in more peripheral regions, the same process may lead to increased costs. Any sizeable positive impact occurring in these regions is likely to be seen over the medium to long-term as advantages of locating in less central regions outstrip the agglomeration benefits of a more central locations. The structural change brought about by advances in telecommunications technology not only has an effect on institutions and organisations, but also on the way individuals live their lives. This has led to use of the term "Infor­ mation Society" to encompass the broader aspects of this process. A dimension of increasing importance in the "Information Society" is that of telework which enables employees and independent workers to work in remote locations using the telecommunication network to receive and transmit data, market their products, search for clients, etc. An increased focus on these broader aspects has led to the EU playing a highly visible role in this area. However, infrastructure and services are mainly financed by the private sector and Structural Fund expenditure has remained modest, amounting to less than 3% of total telecommunications Transversal evaluation of impacts on intervention priorities investments. Nevertheless, in lagging regions it has played an important role in the development of modern telecom networks. In the case of telecommunications Trans-European Networks, the role of the Commission is that of sharing financial risk by providing support for econo­ mically beneficial projects which are of common interest but uncertain or inadequate in terms of financial profitability. Support is provided through a number of instruments: • co-financing of feasibility studies and other technical measures; • subsidised interest rates; • contributions to loan guarantee premiums. • Some key issues for evaluations in the domains of transport and tele­ communications A large proportion of Structural Fund expenditure in these two domains has tended to be concentrated in support measures for infrastructure projects, especially in the case of transport. Whilst this situation is likely to continue in the foreseeable future, the increased interest in "software" transport solutions and Information and Communication Technologies (ICT) applications deve­ lopment means that less tangible effects will need to be increasingly taken into account when evaluations are performed. These two domains have also been the subject of deregulation policies that are having, or will have important structural repercussions. The identification of synergies (either negative or positive) between liberalisation policies and regulatory frame­ works and interventions with explicit cohesion objectives represent an important element of evaluation. Another important aspect of infrastructure projects, relating to cohesion, is their effect on accessibility to peripheral locations. Infrastructure projects are generally large and relatively few in number. They usually concern projects that either take the form of the construction or modernisation of a link between a number of nodes (e. g. roads, railways, cables, pipes, etc. ) or are nodes themselves (e. g. ports, multimodal exchanges, and telephone exchanges). One way of establishing the relative importance of infrastructure projects prior to selecting a number of them for more detailed analysis (i. e. for inter­ mediate or ex-post evaluation) is, for example, to rank them in terms of the size of the intervention in budgetary terms. The importance of TEN projects, Evaluating impacts of interventions in the domains of transport, for example, is reflected by a rating system which identifies priority projects and other projects according to their size. This approach is not, however, necessarily sufficient to comprehend the full importance of a project. Other factors that could be taken into account when ranking projects include social and financial benefits of the project, and the participation of the private sector. Since each project is a component of a system, it is also important to examine their strategic importance within the system itself. It may be possible to do this by examining the importance of the corridor on which the project is situated, in relation to other corridors in the region. Similarly, it may be possible to classify nodes according to their strategic importance in the system. Interventions in transport infrastructure occur at the start of a long causal chain. Because of the distance between an intervention and its eventual socio-economic impacts (i. e. at the level of employment and GNP growth), is difficult to establish clear causal links between this type of project (i. e. the outputs of an intervention) and these global impacts. Evaluations should therefore concentrate on tracking outputs, results and specific impacts of interventions. Furthermore, because medium to long-term effects of infra­ structure projects on cohesion will probably to some extent contradict the short term effects (i. e. over time economies of agglomeration may become negative and relative prices will favour locations outside central poles of activity), the timescale of evaluation is of the utmost importance if the full impact of an intervention is to be observed. In addition to the direct effects that supported infrastructure projects produce, they also give rise to indirect or induced effects by, for example, reducing constraints on the economic activity of indirect addressees. In this way they can give rise to supply-side effects by making productive struc­ tures more efficient (e. g. by reducing delivery times) and reducing costs (e. g. by reducing distances, travelling times, etc. ). They also stimulate demand (demand-side effects) by mobilising human and material resources during the construction period. While Volume 2 of the MEANS Collection presents an information sheet devoted to indicators for the domain of transport infrastructure, by way of an example, a system of indicators that could be used to track the production of impacts arising from an upgrading project for a railway line is presented in Box 36. Transversal evaluation of impacts on intervention priorities Box 36 - Example of a system of indicators for upgrading of a railway line for freight operations Programme Indicators (Related to the Intervention and its effects) Context Indicators (Related to the assisted area) Output indicators Progress Quantity Quality Result indicators Speed Impact indicators Traffic flow Transport system Indirect economic effect Compliance with project duration Rate of completion Number of km upgraded % of line allowing speeds of at least a desired threshold value (i. e. 100 km/h) % of line length allowing trains of heaviest standard load Typical commercial speed of freight trains in that line Average speed on regional lines Freight traffic flows on the line (Tonnes / year) after one year Traffic flow of freight using the new infrastructure and withdrawn from road (Tonnes / year) Traffic flow to and from region (Tonnes / year) Modal distribution of traffic flows within the region (Tonnes / year) % of regional managers who declare that rail freight accessibility ¡s a major constraint Source: Volume 2 of the MEANS Collection Interventions in telecommunications infrastructure are also difficult to link up to socio-economic impacts. In this case, evaluation at the impact level will normally focus on the increased use (e. g. penetration rate, changes in tele­ communications traffic, etc. ) of teleservices and equipment by enterprises and individuals, dueto improvements in the network. Where telecommunica­ tion services supported by Structural Fund interventions are concerned, these tend to be of an experimental nature (e. g. support for the setting up of a telework centre, support for on-line education services, etc. ). This means that as well as looking at any specific socio-economic impacts that they may produce, the survival of projects is an important evaluation issue. While Volume 2 of the MEANS Collection presents an information sheet devoted to indicators for the domain of telecommunications, an example of indicators that could be used to track impacts produced by the deployment of a broadband Hybrid Fibre Coax (HFC) network is presented in Box 37. Evaluating impacts of interventions in the domains of transport, Box 37 ­ Example of a system of indicators for the deployment of a broadband Hybrid Fibre Coax (HFC) network Programme Indicators (Related to the intervention and its effects) Context Indicators (Related to the assisted area) Output Indicators Progress Quantity Result Indicators Network performance Services supplied Impact indicators Connections % of HFC kms in the total network (In the considered region) % of buildings / apartments passed Compliance with project duration Rate of completion Number of HFC kms laid Number of buildings/apartments passed Bit rate Improvement Number of network failures Number of services Number of TV channels Hours of broadcasting supplied (through the network) Number of households that have connected their equipment (TV, PC) to the HFC network (after 6 months) Growth of PC and / or CATV penetration in households in the considered region ICT usage Number of subscriptions to the services Total hours of connections / month Growth in teleservices adoption rate In households in the considered region Source: Volume 2 of the MEANS Collection Given that the main actors in the domain of telecommunications are in the private sector and that some potentially profitable projects, which might have been financed exclusively by private actors, may be assisted by Struc­ tural Funds, the major preoccupation in estimating net effects is the need to identify deadweight (i. e. those investments that would have taken place in the absence of the intervention and their associated effects). ■ Tourism • Structural interventions in favour of tourism The objective of interventions in this domain is to support the diversification of regional economies away from excessive dependence on agriculture and, to a lesser extent, on declining industries. However, while tourism has the capacity to play an important role in supporting socio­economic cohesion, it is generally characterised by a number of structural problems: Transversal evaluation of impacts on intervention priorities • tourism demand is highly seasonal, leading to infrastructures being under­ utilised for substantial periods of the year, and to seasonal / casual employ­ ment patterns; • value added generated by tourist activities is often weak, leading to low profitability; • product quality is highly variable; • levels of training are relatively low; • transport access is a limiting factor to tourist development. Furthermore, the development of tourism needs to be managed in such a way as to ensure its sustainability (i. e. by concentrating on activities that do not produce negative environmental impacts and by mitigating any negative impacts that do occur). Interventions in the domain of tourism are therefore essentially aimed at resolving these problems by, for example, supporting the provision of infra­ structure, widening the range of tourist products, and so on. Interventions, such as support for the construction of conference facilities, can contribute to reducing seasonality by promoting all year round conference tourism. Similarly, the development of new attractions can lead to reduced pressure on established tourist sites during the peak season. This type of intervention also has an impact on transport access (e. g. by smoothing out seasonal travel needs) and on product quality and profitability (e. g. by encouraging clients who demand higher quality products and services). Since tourism is a labour intensive activity relying heavily on locally supplied products and locally based services, it is a domain in which development is largely endogenously generated and which has potentially important effects on employment and living standards, and thereby on the promotion of socio­ economic cohesion. • Some key issues for evaluations in the domain of tourism Although certain interventions support relatively large and visible infrastructure projects, a large number of interventions in this domain are fragmented, hete­ rogeneous and relatively small (e. g. support for improving tourist accommo­ dation, support for museums, heritage centres, training courses, etc. ). The development of tourism is heavily dependent on qualitative factors. Many tourists are discretionary visitors (i. e. they are not constrained by Evaluating impacts of interventions in the domains of transport, factors such as business and family needs to visit a particular region), which means that they are highly sensitive to the quality and price of the product or service they buy. The fact that tourism has traditionally had the status of a commodity is reflected in its lack of variety and its variable quality (as well as low profitability and job quality, if measured in terms of low training levels and the casual nature of employment). Evaluation of interventions that aim to support the development of tourism must therefore take these qualitative factors into account. In this domain it is highly probable that a number of projects would have been undertaken In the absence of Structural Fund interventions (e. g. hotel modernisation, golfcourses, campsites, etc. ). The calculation of deadweight is therefore important when estimating net effects. Since one of the objec­ tives of interventions in tourism is to develop new attractions to reduce pressure on established tourist sites during the peak season, substitution is often an aim of interventions. On the other hand, visitors are only able to visit a limited number of sites during a stay. This implies that assisted sights may benefit at the expense of unassisted sites. It is therefore necessary to be aware of possible displacement effects. By way of an example, a system of indicators for tracking the creation of impacts by interventions supporting the creation of tourist facilities is presented in Box 38. Transversal evaluation of impacts on intervention priorities Box 38 ­ Example of support for the creation of tourist facilities (museums, theme parks, etc. ) Programme Indicators Context Indicators (Related to the intervention and its effects) (Related to the assisted area) Output Indicators Activity of the operator Number of contacts with potential addressees Number of addressees Number of assisted economic units Total number of tourist facilities Capacity Maximum number of visitors per day Result Indicators Duration of visit Time normally required to visit the facility (hours) Cost of visit Average cost per visit of the facilities (euro per person) Impact indicators Fréquentation Attractiveness to Tourists Value added Jobs created Number of visits per year for supported facilities % of visits made by tourists from outside the region Value added generated in euro per year Value added generated In the tourist industry In euro / an Net jobs created (% of which are occupied women) Number of jobs in the tourist industry Source: Volume 2 of the MEANS Collection A more detailed treatment of indicators and other examples relating to tourism are provided in Volume 2 of the MEANS Collection devoted to indi­ cators for monitoring and evaluation. ■ Examples of tools for evaluating structural interventions in t he d o m a i ns of transport, t e l e c o m m u n i c a t i o ns and tourism Transnational evaluations in the domains of transport and telecommunica­ tions have tended to be approached in a top­down fashion, relying heavily on documentary research and secondary data complemented by interviews with key actors, expert reports and, in some instances, case studies and the scores and weights technique. Evaluating impacts of interventions in the domains of transport, The objective of this section is to present a small number of examples of tools used in these domains within the framework of transnational thematic evaluations. A more detailed treatment of evaluation tools is provided in Volume 3 of the MEANS Collection. • SWOT: Example of the strategic overview of transport issues for the Republic of Ireland SWOT analysis (Strengths, Weaknesses, Opportunities and Threats) is parti­ cularly useful for performing a strategic analysis at the time of the concep­ tion or evaluation of an infrastructrure project. Its implementation involves a number of steps: 1. Scanning the key issues and problems faced by a region; 2. Ordering priorities; 3. Identifying opportunities and threats (factors outside the project, region, etc. ); 4. Identifying strengths and weaknesses (factors related to the characteris­ tics of the project, region, etc. ); 5. Synthesising results and analysis of the SWOT matrix; 6. Formulating strategies. In this particular example a SWOT analysis was undertaken to examine the position of the Republic of Ireland with regard to the national transport sector and in relation to the overall context in which it exists. The analysis, based on the factors summarised in the matrix presented in Box 39, comes to the conclusion that the overall position of transport for the Republic of Ireland is one of weakness and inadequacy. This analysis allows the elabora- Box 39 - Example of a SWOT analysis matrix Strengths Weaknesses Industrial base relatively modern Tradition of open economy Educated workforce Unspoiled environment Proximity to sea Political stability Small size, peripheral location Poor indigenous resources Quality of domestic infrastructure Unsatisfactory access to transport Limited finance Unemployment, social problems Opportunities Threats SEM - access to mainland Role as entrepot for the USA? Shannon as European air hub? Suited to tourism and leisure Telecoms based services Mariculture, natural energy European centre moving eastwards Uncertainty of future EC funding Scotland and Wales, post Channel Tunnel Large industrial alliances Unstable Europe- social / economic Emigration of workforce Transversal evaluation of impacts on intervention priorities tion of strategy options aimed at eliminating identified weaknesses or at compensating for them through the development of identified strengths. • Scores and weights: Example of the "Corridor Ranking Framework" (Greece, Ireland and Portugal) This tool is often used in the ex ante appraisal of projects, using a standard set of selection and ranking criteria. It usually involves two stages: 1. Screening out projects that do not satisfy the relevant criteria; 2. Ranking projects that have been retained according to the criteria. In the case of the "Corridor Ranking Framework", presented here, the relative importance of corridors depends on the specific objectives that underpin transport infrastructure interventions and that contribute to attaining the global objective of socio-economic cohesion. The ranking of corridors in this three-country evaluation was undertaken on the basis of seven criteria that were themselves ranked according to their relative importance. The criteria used were the following: 1. Does the corridor provide linkages to the rest of the European Community? 2. Does the corridor connect important population centres (more than 50,000 inhabitants)? 3. Does the corridor provide links to external access points (i. e. airports and seaports)? 4. Are traffic volumes on the corridor high and, as a result, does congestion occur regularly, etc. ? 5. Does the corridor complete land and / or sea routes? 6. Does the corridor facilitate international traffic? 7. Does the corridor integrate remote or isolated regions with development potential? In the case of evaluation of the transport sector in Greece, corridors between economic centres were ranked according to these criteria reflecting a number of transport-related objectives (e. g. improving connections to other European countries, linking up centres of high population concentration, improving links to ports, etc. ) The method of scoring allows the relevance of the project to be taken into account by indicating whether the criteria apply to the project fully, partially or not at all. The criteria are also ranked according to their overall importance. This information is presented in the form of a matrix, an extract of which figures in Box 40. Evaluating impacts of interventions in the domains of transport, Box 40 ­ Extract from a 'corridor ranking framework" applied to Greece 1 Trans­Euro­ pean Road • O • O o Potential candidate corridors for develop­ ment 1) Athens ­ Patras 2) Athens ­ Thessaloniki 3) Thessaloniki­ Igoumenitsa 4) Corinthois ­ Kalamata 5) Igoumenitsa Patras 2 Connects population concent­ rations (cities > 50k) • • loannina 3 O Decreaslr. apriority. 3 Links exter­ nal access points 4 Has traffic volumes 5 Completes land / sea routes 6 Facilitates International traffic • • O O • • • O O o O O • O o O O o o o ■w. 7> Inte­ grates remote / isolated areas O O • • • • Criteria fully applies φ Criteria applies partially O Criteria does not apply It is important to note that this example Is only one way of ranking projects. Another set of criteria would reflect another set of objectives or a different type of intervention (e. g. total project cost, financial return, socio­economic value, relationship to objectives, leverage effects, timing considerations, local geo­political factors, environmental safety and other related objectives, network effects, etc. ). • Case studies: Example of transport sector projects (Ireland) Case studies were employed in Ireland to obtain both quantitative and quali­ tative data on the economic impacts of a number of specific transport projects. Surveys of users (local companies, road hauliers) and cost­benefit analysis were used to inform the case studies. Whilst the case studies selected (a by­pass, a motorway, regional airport development and a seaport) provided some information to complement macro­level data resulting mainly from secondary sources, their contribution to the evaluation could be considered as less than satisfactory for a number of reasons: the benefits arising from the projects were considered indivi­ Transversal evaluation of impacts on intervention priorities dually and therefore could not be fully appreciated because the projects form an integral component of a system; and, secondly, competing factors overwhelmed the specific effects arising from the individual projects. Although case studies in this example provide useful illustrations of interven­ tions and their effects, their contribution to the evaluation could have been increased if a number of methodological considerations had been taken into account. Firstly, the selection procedure for cases should be transparent and the importance of the case in relation to overall policy objectives needs to be clarified. Secondly, as part of a transnational evaluation, the selection of cases should allow for the greatest possible degree of comparison across countries and regions to be undertaken. Finally, the measuring of impacts at the project level neglects the synergies and externalities generated by an intervention. In order to take these factors into account, a case study should focus on a number of key projects within a circumscribed geographical area. • DELPHI: Example of forecasting the development of transport tele­ matics technologies The advantages of telematics in the domain of transport is that they provide a way of optimising the use of existing networks through better traffic mana­ gement and information. They therefore represent a partial alternative to road infrastructure investment. In this particular example, the DELPHI technique was used to forecast the likely level of use of a selection of transport telematics technologies in the year 2015 in medium-sized European cities. The DELPHI technique allows experts to collectively perform quantitative estimations of potential impacts of interventions. The study itself involved a total of 224 experts in Advanced Transport Tele­ matics technology (ATT) from the vast majority of Member States. It focused on 18 different ATT technologies and their possible adoption over the period up to 2015. By participating in a consultation process, this technique allowed the experts to provide information on the technologies they thought were likely to be adopted, the factors that might lie behind their adoption, and those that might restrict or retard this process. A technique related to DELPHI, the matrix of cross impacts (SMIC), was used in parallel with this study. This technique, unlike DELPHI, allows inter­ actions among events to be considered by refining the probabilities relating to the occurrence of individual future developments. The resulting probabili- Evaluating impacts of interventions in the domains of transport, ties can be used directly in the planning process or as the basis for develo­ ping scenarios that can subsequently be used for planning. In total, 126 experts were involved from across the EU. Six possible events were identified which could be broken down into two main groups: • events involving the use of technologies to reduce transport problems • events involving traffic and transport problems or consequences). A first stage of analysis assessed the single probability of the occurrence of the six events considered separately. A second stage examined the condi­ tional probability of the events forecast. The result of the study was a selection of scenarios chosen on the basis of the estimated likelihood of their occurrence. IV Framework for performing thematic evaluations Thematic evaluations can be performed within the framework of a single programme or across a number of programmes. For regional or national level conclusions to be aggregated and compared at European level, evaluations have to be similar in the way they are structured so that they produce results that are relatively homogenous. The first chapter of Part IV presents a stan dardised procedure which, although flexible enough to be adapted to indivi dual evaluation situations, is sufficiently robust to ensure that this objective is attained. Firstly, the second chapter examines the basic principles of transnational thematic evaluation, a number of organisational issues associated with this type of evaluation and how they are used to highlight best practice. There is, however, another approach to undertaking transnational evaluations which is to move away from transnational comparisons, based on synthesising what are essentially a number of individual country or regional level evaluations, towards a multi-country or pan-European impact assessment approach. The future development of transnational evaluations, with an emphasis on moving towards this new perspective, is discussed. The chapter concludes with a brief overview of three evaluation tools that are suited to this approach. A step­by­step approach for thematic analysis The methodological approach proposed below is intended to provide a practical model for assessing, in a way that is relatively standardised, the impacts produced in a particular priority or domain. The need for a standar­ dised approach is particularly important in the case of structural policies because programmes have to be monitored and evaluated throughout the European Union and, as far as possible, the results aggregated. This method can be used to visualise and specify the results of a programme and to produce sufficiently coherent information to enable the Commission services to synthesise data from a number of programmes. ■ A "step­by­step" approach for a thematic evaluation On the basis of the different elements described in this volume, it is possible to establish the different steps required in a thematic evaluation at programme level. This six­phase procedure, comprised of eleven different steps, is set out in Box 41. In the case of an ex ante evaluation, the evaluation team is required to use the first six steps of the procedure. These correspond with the following three phases: preparation of the evaluation, strategic assessment and screening of the programme (screening involves the examination of projects and programmes to determine the degree of impact assessment which will be required). In cases of intermediate or ex post evaluation, all the steps are appropriate. In this instance, the thematic assessment matrix will be initially used to focus the evaluation and secondly to synthesise the results of the analysis. The greater part of the work recommended in this approach should in any case be carried out in the framework of the evaluation of a programme as a whole. Thus, it does not necessarily concern additional work, in so far as the steps described are not intended only for the assessment of the theme. They also serve as a reminder, while the evaluation is underway, to produce results on this theme. Transversal evaluation of impacts on intervention priorities Box 41 Transversal evaluation of a theme : "Step by step" approach for a programme Steps Outputs Preparation • Examine the programming documents. <1 > Identification of the peculiarities of the • Interview the programme managers and stakeholders. programme in relation to the theme: specific objectives, % of budget allocated to specific measures. • Collect available data (time series, Information drawn from the monitoring system, etc. ) and record existing documentation (literature). • Analyse available statistics. <2> Assessment of the most Important Issues in the region in relation to the theme. Reference situation or context analysis. Information concerning programme outputs and results. Strategic assessment • Detail the transversal effects of the programme in relation to the theme according to a typology of impacts. <3> Detailed list of potential effects organised according to the generic criteria selected. • Complete the "thematic assessment matrix" (see Box 42). ■:4> Classification of measures by category: favourable, unfavourable or neutral effect. Screening • Prepare a score card for important impacts assessing the effects of measures (see Box 43). <5> 10 to 20 score cards provisionally completed for the most interesting measures. • Consultation with programme managers and stakeholders. <6> Selection of measures for further analysis. • Collect and analyse information by means of the minimal case study technique, Interviews and / or focus groups. Scoping <7> Score cards completed on the basis of the analysis of selected measures. Programme level synthesis • Complete "summary table" (see Box 45) for each type of impact. <8> Summary tables providing a transversal view of impacts. • Detail the thematic assessment matrix. ^9> Overall view of the value of programmes on the basis of types of impacts defined. Conclusions • Examine the progress made: success and <10> weaknesses of the programme. Useful conclusions for reviewing for the promotion of a priority or interventions in a domain. • Forward conclusions to the Commission <11 > services in a standardised format. Comparable view of the progress made in the EU in respect to the theme. A step­by­step approach for thematic analysis This above procedure is neither exhaustive nor exclusive. Nor is it intended to be a rigid framework. Evaluators have to determine just how far this approach is applicable to the programme in question and, if necessary, add descriptive comments and appropriate indicators. It should be noted that this procedure is not sufficient for the evaluation of the effects of specific measures in a domain: the screening (the identification and selection of measures for more detailed analysis) and scoping (the detailed analysis of impacts) of measures, in particular, needs to be further developed. This procedure is detailed below, using the priority of equal opportunities between men and women as an illustration. ■ Initiating the procedure (steps 1 & 2) Step <1 > is intended to ensure that the evaluators have in­depth knowledge of the content of the programme concerned, including its objectives and information considered to be relevant. A thematic objectives tree can be drawn up, highlighting the objectives of the priority. This phase implies obtaining and consulting all the programming documents and interviewing programme managers. Based on the typology of potential effects described in the chapter on the promotion of equal opportunities, it consists of: • assessing the extent to which the chosen objectives include this priority and, where relevant, are consistent with EU policy, and checking whether the strategy is relevant from this point of view; • analysing the content of the programme in order to assess the extent to which the promotion of equal opportunities has been taken into account during the programming stage; comparing financial grants measure by measure, bearing in mind their specific (targeted) or general nature; making an inventory of the targeted publics of the different interventions in order to identify those interventions which explicitly aim at the priority. It is then essential to define the context indicators which measure the extent of the needs to be met or problems to be solved. Within the framework of ex ante evaluation, they help to show the raison d'être of interventions and to clarify the strategy to be chosen. They serve as a medium for the quantifica­ tion of clear and realistic objectives. Within the framework of intermediate or ex post evaluation, context indicators furnish information about the extent to Transversal evaluation of impacts on intervention priorities which the target has been reached, or the extent to which the needs in the region have been met. In step <2>, the problems encountered by the public (or a geographical zone) concerned are diagnosed, and the potentialities to be developed in the specific national or regional context are identified. A list can thus be drawn up of the main challenges to the effective promotion of equal opportunities. For this purpose it will be necessary to collect literature and have access to databases in order to analyse the national and regional socio­economic context and, more particularly, identify the expectations of potential benefi­ ciaries and political stakeholders. ■ M a k i ng a strategic a s s e s s m e nt (steps 3 & 4) Step <3> consists of a thematic assessment of the programme measures. This implies the classification of each measure in terms of its potential impact on the promotion of equal opportunities. It is necessary first to draw up a typology of impacts. This can be done by drawing up a list of impact descriptors, based on the preceding contextual analysis, and grouping them according to the similarity of impacts (see Volume 4 of the MEANS Collec­ tion for a description of the use of the "impact mapping" technique in this respect). In the case of the promotion of equal opportunities between men and women, the types of impacts selected are: "Living conditions which meet women's needs", "Access for women to the job market", "Women's situation at work" and "Participation of women in the creation of economic activities". The thematic assessment matrix (see Box 42) of the contribution of structu­ ral programme interventions to objectives relating to this priority can be pro­ visionally completed in step <4>. In this way the most promising measures can be identified in terms of potential impacts. The use of this matrix allows the classification of interventions in terms of whether they have a favourable, unfavourable or neutral effect. Thus, it is possible to identify interventions that warrant further analysis during the phases of screening and scoping. Finally, the matrix enables the identifica­ tion of those objectives mentioned in the programme but which few measures help to achieve. A step­by­step approach for thematic analysis Box 42 ­ Thematic assessment matrix. Example relating to equal opportunities Type of impact expected / achieved on the promotion of equal opportunities Area of intervention (Measures) More appropriate living conditions Access to the job market Situation at work Participation in the creation of socio­ economic activities Infrastructures Measure N°: Measure N°: Human resources Measure N°: Measure N°: Productive environment Measure N°: Measure N°: Added value of the programme In terms of equal opportunities Provisional assessment of impact / /? ) X Beneficial effect Effect probably No significant Negative effect beneficial effect ■ Screening and scoping (steps 5, 6 & 7) Step <5> concerns the identification and selection of measures for further analysis. It requires the creation of a score card for each measure identified as having a potential impact on the promotion of equal opportunities. An example is provided in Box 43. In the preparation of the score cards the following types of impact should be covered: • positive or negative; • direct or indirect; • temporary or permanent; • short or long term. Transversal evaluation of impacts on intervention priorities Box 43 - Score card. Example: assessing the extent to which a measure takes into account of the priority of equal opportunities Sub-programme / priority: Measure (title): Objective: Budget (% of funding by the EU): Output: Potential positive impact of Significant bene- Hardly signifi- cant beneficial effect projects/actions on: ficial effect No significant effect Negative effect Living conditions better suited to women's needs Women's access to the job market Situation of women at work Participation by women in the creation of socio-economic activities 1 1 1 0. 5 0. 5 0. 5 0. 5 0 X X X X Total number of points | | If the measure has a direct or explicit impact In terms of equal opportunities, multiply the score by 1. 5 Give details on the two most significant positive or negative effects: Secondly, the evaluators are advised to consult with programme managers and stakeholders about the results obtained during the first two phases of the evaluation (step <6>). The objective of this consultation is to select key indicators and the measures that are to be the subject of further analysis. Until now there has been no attempt to quantitatively measure the changes that have occurred in relation to the priority. At this stage, the information summarised in the thematic assessment matrix and on the score cards should be complemented by data gathered by the relevant statistical agencies on the indicators selected. After the score cards have been the subject of consultations, the measures most likely to have a significant impact on the promotion of equal opportuni­ ties are selected for analysis (step <7>). This analysis involves an examina­ tion of impacts using minimal case studies, interviews and / or focus groups (see Box 44). This technique, based on the limited analysis of impacts in order to consolidate the assessment of impacts carried out on a strategic basis, is known as scoping. Cross programme thematic evaluation Box 44 - Interviews, focus groups and case studies Individual interviews can be used for all types of evaluation where the information needed is more complex than that which can be conveyed by questionnaire. Interviews range from being relatively informal and unstruc­ tured to highly formalised, based on pre-determined questions. They are particularly useful for describing procedures and processes (e. g. the implementation of a programme) and the likely impacts of an intervention or impacts that are difficult to quantify, such as behavioural changes or intangibles. Group interviews can largely be considered as a number of simultaneous individual interviews. Focus groups actively seek to harness the dynamics of the interactions between the individual participants and to integrate it into the analysis of complex issues. The interaction among participants is particularly useful when opinions about a certain issue are not fully formulated. The technique can therefore be used to identify the needs of a region, clarify the objectives of interventions, examine the impacts of innovative measures, focus or refocus the implementation of a programme, or interpret the impacts of an intervention. Case studies are used for the analysis of specific cases, ranging from an individual to a geographical region. They can also be used to analyse decision making processes. Both interviews and focus groups can be used as data gathering tools within the framework of case study. Case studies are particularly useful for making detailed analyses of the complex process of programme implementation and of their impacts. They are often undertaken to verify causation or to qualitatively test and validate measurements of effects obtained by other tools. This tool can also be used when there are very few units of analysis by matched pairs of similar units. This involves finding two units of observation - one of which is an addressee of a programme and one of which is not - that closely resemble each other (e. g. SMEs in the same sector, with the same number of employees, etc. ) and that are in a similar context (e. g. in a declining indus­ trial region where unemployment is high, etc. ). The relative evolution of the units can then be studied. Transversal evaluation of impacts on intervention priorities ■ Synthesising the results (steps S & 9) The analysis by measure is essential for the construction of an evaluation of the programme as a whole. Evaluation results must also be produced at programme level, then at the higher levels. To globally assess the effects of measures at the level of each programme, a summary table (see Box 45) can be prepared for each of the four main types of impact, identified as being relevant In the case of structural interventions (step <8>). The summary table will be completed using: • the quantitative assessment provided by the score cards (far right-hand column of the matrix); • the analysis, providing both quantitative and qualitative data, performed during the scoping phase (cells of the matrix). Box 45 - Summary table of measures and their effects. Example of improving the suitability of living conditions to women's needs Summary table: synthesis of impacts in terms of living conditions better suited to women's needs Measures with a significant impact Budget (% of funding by the EU) Outputs Results which were expected and / or produced Impacts which were expected and / or produced N°: (title) N°: (title) N°: (title) N°: (titre) Total points or X ( in the case of a negative impact (source: score cards): General comments this type of impact: for Since the Commission will need to have an overall view of the impact of interventions on the priority of promoting equal opportunity between man and women, it may be useful to prepare a brief, very general summary on the Cross programme thematic evaluation (positive and negative) impact of each programme (<9>). The thematic assessment matrix, previously completed on a provisional basis can be used for this purpose (see Box 46). The matrix approach is sufficiently simple (without being simplistic) to meet practical needs. However, it remains necessary to analyse available indicators, in particular as far as context is concerned. The matrix approach makes it possible to achieve the standardisation needed for an overall view at European Union level. It serves to simplify an extremely complex reality and to present it in a form which is synthetic and comprehensible at policy level. The key steps and evaluation tools employed in this approach are summari­ sed in a highly simplified example in Box 46. Four measures initially enter the strategic assessment phase, of which three are considered sufficiently important to be retained for further study. At the screening phase Measure 4 is eliminated which leaves Measures 1 and 3 to enter the scoping phase. The conclusions of the analysis made at the scoping phase are presented in a summary table and this is then used to complete the thematic assessment matrix. Box 46 ­ Summary of the key steps and evaluation tools Thematic assessment matrix Screening Step <4> and Step <9> Step <5> and <6> Scoping Step <7> Measure 1 2 3 4 k f , ) , I «* • 1 3 > 4 ­ * ·! 1 ' ­ * ¡ 3¡ _l S u m m a ry t a b le (one per t y pe of i m p a c t) S t ep < 8> Measure Resources Outputs Results Impacts 1 τ Transversal evaluation of impacts on intervention priorities I Concluding on the merits of t he p r o g r a m me as regards the theme evaluated (steps 10 & 11) The conclusion of the evaluation report (step <10>) will obviously include recommendations on the best way of taking the evaluated theme into account when formulating objectives, programming and selecting projects. So that a priority can be better taken into account by decision-makers, the evaluation team must make significant efforts to: • propose new strategies of intervention; • strengthen the most promising measures; • formulate recommendations for enhancing the potential of the targeted social groups and areas. On a more technical level, recommendations are needed so that in the future the systems of monitoring indicators and of statistics can include information that will provide a better understanding of the situation. Finally, in step <11 > the conclusions provided by the procedure will be suf­ ficiently standardised to nourish debates and comparisons of the promotion of equal opportunities between men and women (or other priori­ ties or domains) at the European level. Cross programme thematic evaluation ■ Basic principles of transnational t h e m a t ic evaluations The preceding chapters have examined the evaluation of a programme from a thematic perspective. Evaluations at the programme level focus principally on the examination of programming undertaken by Member States. Whilst these evaluations allow the appreciation of interventions on a national / regional scale, they do not provide information that allows a Europe­wide analysis of impacts, the appreciation of Community value­added or a pan­ European strategic analysis of Structural Fund interventions. In order to achieve these more ambitious evaluation objectives, it is necessary to undertake thematic evaluations at transregional or transnatio­ nal level. It is important to note, however, that for this type of evaluation to be fully effective the methodology used should go beyond the simple supra­ national synthesis of a number of single evaluations (see Box 47, upper figure). The approach followed should be based on the collection, analysis and synthesis of data within a common and co­ordinated single evaluation framework across all the countries involved (see Box 47, lower figure). Transversal evaluation of impacts on intervention priorities Box 47 ­ Comparison of supra­national synthesis of programme evaluations and transnational evaluation Supranational synthesis of evaluations (a), (b) and (c) J_ Evaluation (a) Evaluation (b) Evaluation (c) Programme (a) Programme (b) I Programme (c) Single transnational evaluation Programme (a) J 1 Programme ! ; (b) : 1 J 1 Programme ! ; (c) : ι I ■ Organisational issues A major problem in commissioning thematic transnational evaluations is the question of how to take into account differing national or regional contexts, thematic specificities and interrelations between different themes and domains. The consequence of these complexities is that the drawing up of the terms of reference for an evaluation of this type is extremely problemati­ cal. It is for this reason that an essential first step is to set up a technical committee composed of experts and academics with relevant sectoral knowledge, and national experts who understand the national and regional socio­economic contexts in which the evaluation will take place. The role of the technical committee is to steer a feasibility study by evalua­ tors of two or three countries (carefully chosen for their representativeness Cross programme thematic evaluation and relevance in terms of the theme of the evaluation). This study will involve the elaboration of a formal pre-evaluation document in which priority questions, programmes to be evaluated, timescale, budget and the evalua­ bility of interventions are addressed (e. g. Is the evaluation politically and technically opportune? Is the budget and timescale adequate, considering the characteristics of theme and the stock of existing knowledge that may be mobilised?). When designing the evaluation the evaluation team should: • draw up evaluatory questions that disregard country and regional factors and concentrate solely on thematic aspects of the evaluation, and which are specifically centred around a European perspective; • tightly define the criteria for the selection of case studies and sampling so as to improve comparability and take into account of country- and region- specific contextual factors; • elaborate an analytical grid which takes competing factors into account. This work should be then validated by the technical committee. Case studies should be selected by the technical committee and validated by the relevant national experts who would judge the pertinence of the cases chosen. The number of case studies performed should be relatively limited and they should be likely to capture strong structuring effects. These struc­ turing effects should be analysed against a contextual backdrop which may be regional, national, European or even international, depending on the par­ ticular cases selected. The output of this process will be a document which is the result of interac­ tions between actors with sectoral and national socio-economic experience. When the final evaluation report is produced it is important that the Member States involved in transnational evaluations should have the opportunity to verify the interpretation of the results of the evaluation. Transversal evaluation of impacts on intervention priorities ■ Highlighting e x a m p l es of best practice Carefully designed and implemented transnational evaluations can comple­ ment existing programme level evaluation by highlighting examples of best practice and allowing experiences to be transferred between regions and countries. The objective of this section is to suggest a number of practical recommendations that are aimed at ensuring that evaluations are fully adapted to the goal of highlighting best practice. Since Community programmes are both complex and open ended, the way in which they are implemented is an important evaluation issue. The search for examples of best practice should therefore examine not only the impacts of programmes but also the process of implementation, and should make comparisons of delivery mechanisms in different countries. In order to find examples of best practice, it is important that populations of beneficiaries, geographical units (regions, sub­regions, towns, etc. ), projects, interviewees, etc. selected for case studies should be similar in terms of the context in which they are situated. The contextual situation in which the selected case studies are embedded can be initially established by a SWOT analysis. Points of comparison also have to be established by which performances can be judged. This can be done using the benchmar­ king technique. The SWOT framework (Strengths­Weaknesses­Opportunity­Threats) can be used to undertake a strategic analysis of a programme and the region in which it is to be implemented (internal factors) as well as the overall policy environment (external factors). This tool can be used to outline an economic development strategy and is therefore particularly useful during the design and ex ante assessment stage of a programme. It can also be used for mid­ term and ex post evaluations to take into account contextual and identify problems of comparability. Box 48 illustrates this principle. Cross programme thematic evaluation Box 48 ­ Sampling with the objective of highlighting best practice Country 1 Country 2 Country 3 SMEs medium­tech 20­50 employees _ Λ_ e. t x e t n o C d. c. b. a SMEs medlum­tech 20­50 employees L Λ 1 I 1 1 1 1 1 SMEs medium­tech 20­50 employees m ι • I I 1 1 1 ■ 1 1 1 1 _. ( · \ V · J Cross­country sample of various practices embedded in similar contexts If the theme which is the object of the evaluation is SMEs, the constitution of a cross­regional sample requires that the firms have similar characteristics (e. g. between 20 and 50 employees in a medium­tech sector of the economy). In order to assure comparability, the firms are drawn from similar contextual situations, minimising the difference in regional context. A direct comparison is then feasible across countries. Benchmarking consists of making a qualitative comparison between one intervention and another, comparable in terms of being embedded in the same contextual situation, which is reputed to be an example of best practice. By doing this, the strengths and weaknesses of the intervention being evaluated can be highlighted. This technique can also be used at a more strategic level to compare the organisation of programmes. ■ Future directions for transnational evaluations Evaluation practice and experience in the field of transnational thematic eva­ luations have grown rapidly and a second generation of this sort of evalua­ tion, with an emphasis on highlighting best practice, is already developing (see Box 49). Transversal evaluation of impacts on intervention priorities While the type of transnational evaluation discussed above is capable of highlighting examples of best practice, it is not capable of providing a picture of pan­European achievements. This sort of study, which needs to take into account different national or regional contexts, could be used to inform European­wide strategic planning, for example, permitting the drawing up of a masterplan for Trans­European Networks or for the production of recom­ mendations and guidelines adapted to Community­wide conditions. Box 49 ­ The development of transnational evaluations 1 ■ Experience of transnational evaluations • mainly based on documentary research, secondary data, and interviews with key actors • scores and weights • SWOT analysis • project case studies Examples: • Transportation • Telecommunications • Tourism $ 2° Experience of transnational evaluations 1° + additional tools including: • country­by­country reviews of existing literature • secondary data analysis • surveys • regional and project case studies Examples: • Business information Centres • Environment • RTD • SMEs ^ 3° Experience of transnational evaluations 1° el 2° + additional tools such as: • research synthesis • expert panels • pan­European surveys • embedded case studies •others. In addition to the range of tools suited to thematic evaluations presented throughout this volume, there are a number of tools are suitable for underta­ king pan­European analysis. These tools are presented below. • Multi­country / multi­regional econometric analysis When surveys can be undertaken, the data collected can be used to construct ad hoc multi­country or pan­European econometric analysis of impacts. The results furnished by this type of analysis can transformed into coeffi­ cients which can subsequently be used to make a projection of the results Cross programme thematic evaluation for either particular aspects of an intervention or for its overall projected effects. The use of coefficients also requires monitoring data to provide information on the addressees of interventions and the distribution of expen­ diture, etc. Impacts can be analysed both in a cross-sectional perspective (i. e. a variety of interventions at a given moment in time) and over time (i. e. using time-series data). The use of this tool requires that sampling be correctly undertaken. Unlike the sampling technique used when benchmarking is the objective of an eva­ luation, in this case samples should be constituted from across a diverse range of country and regional contexts. In order to do this, a common, co­ ordinated methodological approach has to be used which involves carefully controlled data collection to ensure that the selection of units of analysis features the level of representativeness required. This approach is illustrated in Box 50 where careful sampling allows a cross-regional sample, drawn from a variety of different contexts, to be constituted. Box 50 - Sampling with the objective of making a pan-European analysis Country 1 SMEs medium-tech 20 - 50 employees Country 2 SMEs medium-tech 20 - 50 employees Country 3 SMEs medium-tech 20 - 50 employees I 1 1 > | 1 1 1 t x e t n o C e. d. c. b. a 1 ' 1 t 1 1 1 1 i 1 f » • \ · y f / • V · · · Transversal evaluation of impacts on intervention priorities Whenever feasible, comparative analyses should be performed with non- European countries (e. g. the United States, Canada, Australia, etc. ). • Multi-site case studies with replication design Case study analysis is a useful tool for analysing socio-economic phenomena that are difficult to separate from their contextual conditions. The unit of analysis of the case study may be an individual, an organisation, a process, an industry, a region, etc. The main unit of analysis (e. g. a region) may furthermore contain sub-units of analysis (e. g. firms, infrastructure projects, etc. ). Case studies can either be single or multiple. The use of multiple case studies with replication design is a particularly important technique for strengthening or broadening analytical generalisa­ tions. This type of case study is based on a series of waves (usually two or three) of case studies carefully chosen to support a given proposition. Depending on the design of the evaluation the case studies will produce either: • similar results providing corroborative evidence; • contrasting results but with similar theoretical underpinnings (i. e. the theory is corroborated); • both replicated and contrasting results within the same study. If, however, the results produced in some of the cases are contradictory in some way, the initial proposition cannot be accepted. This may cause it to be retested, by means of a second and, when necessary, a third wave (see Box 51 ). Example: Providing advice to city administrations In a study designed to examine attempts by universities and research groups to colla­ borate with local government officials, four different sets of replicated case studies were undertaken (see Box 51). The articulation of the propositions supported by each of the different sets allowed an overall conclusion to be reached which was not evident at the outset of the study. An initial group of case studies concluded that uni­ versities and research centres had failed to help city administrations. A second group concluded that this phenomenon was not restricted to universities but included other organisations. A third group showed that in some cases universities had been able to help other types of organisations. Finally, a fourth group showed that some groups had been able to help city administrations by taking implementation issues into account. The overall conclusion was that city administrations had particular needs when receiving advice from outside bodies, a conclusion which differs significantly from the initial one. Cross programme thematic evaluation Box 51 - Example of multiple-case studies with replication design (Wave 1 ) Case studies focusing on universities and their relations with city acministrations (Wave 2) Case studies on relations between non-university groups and city administrations (Wave 3) Case studies on relations betwwen universities and organisations other than city administrations (Wave 4) Case studies on groups concerned not only with new ideas, but also their implementation Source: from Yin R. K. (1994), Case study Research: Design and Methods, London: Sage. • Research synthesis Research or evaluation synthesis is the analysis of existing research using a set of systematic procedures - or meta-analysis techniques - for summari­ sing quantitative results from multiple studies. This technique is based on a systematic and rigorous approach to synthesising past research. Unlike a literature review, research synthesis takes into account contradictory results when analysing the existing stock of knowledge about a subject. Transversal evaluation of impacts on intervention priorities This technique has two benefits: first, by drawing on a number of well designed and executed studies, the resulting analysis is highly robust because based on solid and reliable knowledge; secondly, if research synthesis shows that the existing base of knowledge is not well-founded, it enables decision makers to avoid basing their policy making on incomplete or erroneous information, and provides guidance for future research. The procedure for performing research synthesis is summarised in Box 52. Box 52 - The seven steps of research synthesis Siepi Selection of the questions to be addressed Step 2 Collection of the all relèvent documents and identification of key issues Step 3 Development of screening criteria and screening out of irrelevent material Step 4 Development and implementation of critical asessments of studies Step 5 Redetermination of the appropriateness of this technique in the light of the material reviewed Step 6 Undertaking of the evaluation synthesis Step 7 Reporting the findings of the synthesis Conclusion Transversal evaluation of impacts on intervention priorities The development of thematic evaluations within the framework of the Struc­ tural Funds has been shaped by the evolving policy context within the European Union. As a consequence, the state of the art in this area was initially dominated by experience in the evaluation of employment effects. As the decade has progressed, however, environmental concerns and equality between men and women have climbed the political agenda, and this has been reflected in the accumulation of evaluation experience in these themes. At the same time as this process of accumulation of thematic evaluation experience has been occurring, the need for comparisons between pro­ grammes has become increasingly important, hence the need for transna­ tional thematic evaluation. Experience in this still emerging area of evalua­ tion remains relatively limited for the moment. While an in-depth evaluation of impacts within a particular theme requires corresponding expert knowledge which can only be provided by specialists, the central premise of this book is that, with a basic understanding of a theme's characteristics, an impact assessment can be designed or underta­ ken by evaluation units and evaluators without in-depth knowledge of the theme. It is on this basis that existing thematic evaluation experience and general evaluation expertise have been combined to produce this book focusing on how to assess the impacts of Structural Fund interventions on the four political priorities of the EU and on number of domains of intervention. The approach taken in this book has been to examine thematic impact assessment in such a way that methodological considerations, rather than evaluation themes, are the dominant perspective. This approach has been particularly fruitful since it has given rise to a the construction of an evalua­ tion method - the step-by-step procedure - which draws upon evaluation practices and tools that have been developed within the various thematic frameworks An example of this is the screening and scoping technique which has been mainly used for ex ante environmental impact assessment. Another example of "cross-fertilisation" between themes is that of mains­ treaming, which is finds its way into the step-by-step approach by way of the score card. Moreover, this is a strategy which is particular to the evaluation of equality between men and women, but which seems to be a promising approach for evaluating impacts on SMEs within the framework of interven­ tions aimed at improving the competitiveness of the productive sector. At the conceptual level, the notion of sustainability which can be applied across a wide range of themes. Similarly, the concept of leverage can be applied to Conclusion all types of interventions where the objective is to encourage private sector co-financing. Substitution, displacement and deadweight, concepts origina­ ting from the evaluation of employment effects, are also applicable to other themes. Finally, although a large part of this Volume has focused on drawing on existing evaluation experience, the future of the evaluation of Structural Fund interventions has not been forgotten. It is for this reason that a number of tools are discussed that seem particularly adapted to complementing transnational thematic evaluation aimed at highlighting best practice with in- depth evaluations of impacts at the pan-European level. Appendix Appendix 1 contains an index of tools classified by their thematic suitability in the form of a matrix. Appendix 2 presents the terms of reference for a programme evaluation incorporating explicit instructions relating to the inclusion of Community political priorities: "Intermediate Assessment of the Highlands and Islands objective 1 Programme". Appendix 3 presents the terms of reference of a transnational thematic eva luation: "Evaluation of RTD actions in the case of Structural Funds" (1998). Appendix 4 presents a list of boxes. The documents presented in Appendices 2 and 3 are not intended to serve as strict models upon which terms of reference should be based. They can, nonetheless, be used to guide the drawing up of future terms of reference which the authors should tailor to the specific characteristics of the evalua tion to be performed (e. g. in terms of particularities of the theme, evaluation tools to be chosen, organisational factors, etc. ). Interviews Case Focus studies group Question­ naire SWOT surveys CBA and MCA Scores Screening and and DELPHI weights scoping Expert panels coefficients Input­ Macro­ and output ecowmethc statistical analysis models analysis Multipliers, Environ­ ment Page 54 Page 55 Page 52 SMEs Page 73 Page 75 Page 73 Page 73 Page 73 T3 CD < ω 5' 3 Employ­ ment Equal opportunities Page 101 Page 101 Page 103 Page 104 Page 104 Page 126 Page 125 Page 123 Page 126 Page 123 RTD Page 147 Page 147 Page 147 Page 147 ·­*■ O o_ C/5 03 3 O. O Π) O ^ -\ (Λ •a o =i o. Ξ" en Transport Page 163 Page 161 Page 162 Page 164 re 3 re C/5 Appendix 2: Extracts from a Terms of Reference for a programme evaluation incorporating thematic elements Intermediate Assessment of the Highlands and Islands Objective 1 Programme This document explicitly presents the requirement to evaluate Community priorities in the paragraph on "Specific Objective" of the evaluation. Examine each priority and measure and consider their appropriateness in addressing the aim of the programme. Consideration should be given to the number of measures, their suitability and scope and the continued relevance of targets, indicators and outputs. Recommendations should be brought forward on amendments. Annex A sets out in summary form the key issues which the partnership has identified in terms of implementing the SPD by priority and measure. The final report should address each of these issues albeit in varying detail, and where appropriate, provide advice or recommen­ dations for the future implementation of the programme. Assess performance at programme priority and measure level, identifying cause of variance in terms of commitment and expenditure, having regard to external factors and match funding (comment on the extent to which such factors can be accommodated within the existing implementation). Examine the synergy between the different Structural Funds, between Priori­ ties and between the SPD and Community Initiatives. To comment and advise on linkages within the programme and the extent to which objectives are being achieved in particular job creation, environmental sustainability and community development. Details are given of key issues concerning various priorities PRIORITY 1 : SUPPORT FOR BUSINESS • Is there adequate balance across different measures? • Can different measures and different funds be better integrated? • Are the targets appropriate in relation to the strategy? Transversal evaluation of impacts on intervention priorities Measure 1: Financial support for the development of new ventures and SMEs • Scheme coverage - does this require to be focused? - does it require better co-ordination. • Number of schemes - Scope for more? Capacity for more? • Leverage - Is there a need to establish thresholds? • Does there need to be greater of sectoral focus Measure 2: Financial support for research and development appro­ priate to SMEs • Why are SMEs not more interested in R&D? • Does product development require more emphasis? • How can the measure be made more attractive? [etc. ] PRIORITY 3: PRESERVATION AND ENHANCEMENT OF THE ENVIRON­ MENT • Are the measures too or not sufficiently constrained? • Is the scope of measures too limited? • How can a better integration between-funds be achieved? Measure 1 : enhancement and protection of the environment and mana­ gement schemes • Is the scope of measure sufficient? • Are targets suitable? • How can support be adjusted in relation to the targeted problem? Measure 2: pollution control, waste disposal and the development of recycling systems • How can statutory and non-statutory responsibilities be better reflected in the system? • How can re-cycling be encouraged? • Can the innovatory character of projects be increased? [etc. ] Appendix 3: An example of terms of Reference for a transnational thematic evaluation Evaluation of action in favour of RTD within the framework of the Structural Funds ■ Background Presentation of the theme, related policy interventions, the context in which it is situated and the raison d'être of the evaluation. The importance of research and technological development (RTD) for improving the competitiveness of the productive sector through product and process innovation, especially by SMEs, is recognised in the context of Structural Funds. The Commission's First Action Plan for Innovation in Europe also stressed the need for all Community instruments to be mobilised to support RTD and innovation with an increased support from the Structural Funds through other forms of intervention such as Article 10 of the ERDF and some Community Initiatives (SME initiatives, ADAPT, LEADER II). The build up of a RTD capacity in assisted regions is a necessary but not a sufficient condition to enhance their competitiveness and economic perfor­ mance. The RTD investment effort should be related to the build up of the wider innovation system, in particular through the creation of durable links between the RTD system and the regional industrial structures. Within assisted regions, Community interventions for RTD and innovation under the Structural funds provide an opportunity to strengthen their capabi­ lities and to reduce the technology gaps with the most developed regions. Most objective 1 regions (and to a lesser extent objective 6 regions) face a number of problems relating in particular to the concentration of RTD related activities in the most developed areas, weak industrial participation as well as human resource shortages. In these regions, the provision of facilities and equipment and the availability of researchers are priorities as well as the application of research results and the transfer of innovation and technology to local business and industry. Most objective 2 regions, which often are Transversal evaluation of impacts on intervention priorities endowed with better RTD capacity, focus their actions towards the develop­ ment of specific segments in line with the needs of regional SMEs, and espe­ cially to technology and know­how transfer to local business and industry through regional and inter­regional co­operation networks. Given the complexities of developing RTD policies for both types of areas, it is imperative to have a detailed understanding of the current regional situation in order to determine what policies might be most appropriate. A region­specific analysis, according to characteristics of the objectives, enables current innovation and technology levels to be assessed and gaps to be identified in the policy framework. In view of the preparation of the post­1999 Structural interventions, the Commission is currently preparing a communication on RTD and innovation. Simultaneously, it is envisaged to launch a comprehensive thematic study to feed the reflection on these policy issues ■ Aims and objectives of the study This section of the document should clearly list the central questions to be addressed by the evaluation. While these questions are obviously specific to the theme to be evaluated in terms of the programmes implemented and the impacts likely to be generated, they will be focused on key factors that are generally applicable to the evaluation of Structural Fund interven­ tions: • Impacts to be analysed • Effectiveness, efficiency, relevance and utility of interventions • Coherence and synergies between different levels of intervention (i. e. regional, national and European) • Coherence and synergies between different Community level interven­ tions (i. e. Community Framework programmes, Structural Fund pro­ grammes, Community initiatives, innovative or specific measures) • Future directions for policy The importance of research and technological development (RTD) for improving the competitiveness of the productive sector through product and process innovation, especially by SMEs, is recognised in the context of Structural Funds. The Commission's First Action Plan for Innovation in Europe also stressed the need for all Community instruments to be mobilised to support RTD and Appendix innovation with an increased support from the Structural Funds through other forms of intervention such as Article 10 of the ERDF and some Community Initiatives (SME initiatives, ADAPT, LEADER II). The build up of a RTD capacity in assisted regions is a necessary but not a sufficient condition to enhance their competitiveness and economic perfor­ mance. The RTD investment effort should be related to the build up of the wider innovation system, in particular through the creation of durable links between the RTD system and the regional industrial structures. Within assisted regions, Community interventions for RTD and innovation under the Structural funds provide an opportunity to strengthen their capabi­ lities and to reduce the technology gaps with the most developed regions. Most objective 1 regions (and to a lesser extent objective 6 regions) face a number of problems relating in particular to the concentration of RTD related activities in the most developed areas, weak industrial participation as well as human resource shortages. In these regions, the provision of facilities and equipment and the availability of researchers are priorities as well as the application of research results and the transfer of innovation and technology to local business and industry. Most objective 2 regions, which often are endowed with better RTD capacity, focus their actions towards the develop­ ment of specific segments in line with the needs of regional SMEs, and espe­ cially to technology and know-how transfer to local business and industry through regional and inter-regional co-operation networks. Given the complexities of developing RTD policies for both types of areas, it is imperative to have a detailed understanding of the current regional situation in order to determine what policies might be most appropriate. A region-specific analysis, according to characteristics of the objectives, enables current innovation and technology levels to be assessed and gaps to be identified in the policy framework. In view of the preparation of the post-1999 Structural interventions, the Commission is currently preparing a communication on RTD and innovation. Simultaneously, it is envisaged to launch a comprehensive thematic study to feed the reflection on these policy issues Transversal evaluation of impacts on intervention priorities Scope The field of application specified should delimit the evaluation according to the relevant dimensions such as time, geographic zone, and type of intervention: • On what period should the evaluation focus? Will the evaluation involve a comparison between programming periods? • Which countries / regions will be involved in or excluded from the eva­ luation? • Which policy instruments would be included in the study? ·- Suggest how interventions and their objectives of might be categori­ sed. This will help give evaluators to better orient their proposals. Considering the content and policy relevance of RTD actions and the need to cover all relevant objectives under Structural Funds, the study will be divided into two lots. It will focus on the one hand on objective 1 and 6 regions (lot 1) and on the other hand on objective 2 regions (lot 2), but due consideration will be given where relevant to other forms of intervention such as Community ini­ tiatives and innovative actions under the ERDF art. 10. It will cover both Structural Funds programming periods but particular attention will be given to the 1994-99 period, for which most information is available. The study will also consider the EU Framework Programmes and relevant national RTD policies including significant regional state aids for RTD. Regarding the actions or activities to be evaluated, there needs to be a clear classification of what makes up RTD activities. The coverage will be broader than RTD in the strict sense covering largely the innovation system. The analysis should refer to a common framework, as developed under the Action Plan for Innovation, in order to allow for a more coherent evaluation of RTD and innovation content of Structural Funds. For example, in objective 1, it should be possible to assess the catch-up process in terms of RTD infra­ structure, and to distinguish between scientific investments (research centres, research projects. ) and technological investments (technological parks, technology centres. ). The suggested framework encompasses the following categories: (1) Enhancement of scientific and technological potential • Scientific potential (infrastructure, equipment, scientific actions,) • Technological potential (infrastructure, equipment, technological actions,) Appendix (2) Support to Innovation • Promotion of an innovation culture, which includes human resources deve­ lopment through education and training measures, research mobility, stimula­ tion actions, dissemination of RTD results, promotion of public awareness, etc. • Regulatory and financial framework (legislative and regulatory environ­ ment, innovation financing through seed capital, venture capital or other) • Link between research and innovation , including technology transfer, promotion of networking, strengthening links between enterprises, universi­ ties and research centres In reviewing the different types of RTD innovation actions, it would also be interesting to consider relevant sectors of the Framework Programme such as: ­ information and communication technologies, ­ industrial technologies, ­ life sciences, including bio­technologies and agriculture, in order to assess complementarity and synergy between Structural Funds interventions and the Framework Programme itself. As regards the development of RTD and innovation indicators, the Eurostat "Innovation manual" should constitute an essential source of information for this study. ■ Content of the study This section outlines how the evaluation will be structured. It should present the methodological framework of the evaluation including: • evaluation tools to be used • which methods of data collection will be used • details of the types of analysis to be performed ·■ The methodological framework should be drawn up in such a way as to ensure that the evaluators, while respecting its overall structure, feel able to use their experience and know­how to make practical suggestions on how the evaluation could be performed. Transversal evaluation of impacts on intervention priorities It should comprise the following aspects: * Methodological framework This part will contain the development of a methodology to allow for compa­ rative analysis and measurement of performance of RTD related actions. This should include: (a) the adoption of a thematic approach requiring the comparative and simul­ taneous assessment of common themes and issues (b) the proposal of appropriate indicators, relating to inputs, outputs, impacts, management and economic context (c) the design of selected instruments to assess performance of a sample of RTD related actions (d) the design of questionnaire forms and identification of key informants (basis for field work) (e) the selection and implementation of case studies * A comparative analysis of RTD policies and programmes and the impact of Structural Funds interventions In the context of Structural Funds, the Commission wishes to analyse in a comparative fashion: - t he RTD funding, i. e. the amount and proportion of Structural Funds budget allocated to RTD, and a comparison with national or other Community funds allocated to RTD for each relevant objective, member state and region; - the importance assigned to different RTD related actions and their diversity throughout the regions; - t he quality of RTD organisation and structures, relating in particular to management structures and evaluation procedures; - the effectiveness of the instruments or mechanisms to promote RTD and innovation actions; - the impact on human resources and training for RTD. * Assessment of a sample of RTD related actions at country/regional level (field work) This part will consist of an in-depth analysis of a significant sample of RTD actions funded by the Structural Funds for each of the countries concerned. Appendix The selection of RTD actions will be based on criteria such as financial dimension and strategic importance. Fieldwork will be conducted in relevant countries or regions in accordance with a standardised methodology. • Assessment of Community value-added and best practices This part of the study will evaluate the RTD related actions funded by Structural Funds, and draw lessons for regional policy. Particular attention is to be paid to: - t he quality of RTD actions, and the changes observed between the 1989- 93 and 1994-99 programming periods; - the value added of Structural Funds actions in the field of RTD • the identification of best practices in terms of RTD industry linkages, of technology transfer mechanisms and institutional designs in the assisted regions and analysis of transferability conditions to other regions • the complementarity and synergy within Structural Fund programmes and with the EU RTD Framework Programme as well as with national RTD policies. • Synthesis of main findings and policy issues This part will draw conclusions and recommendations, with special regard to the future of structural policies. Some of the issues at stake have already been addressed in the first Cohesion Report and the current Communication on "RTD, Innovation and Cohesion". Nonetheless, further thought should be carried out on the following questions: • Should there be more RTD activities funded under the Structural Funds? • How should RTD actions be focused? • Have the Structural Funds been sufficiently considered in the Fourth Framework Programme and vice versa? • Should more be done in the future to achieve better synergy of both EU policies? ■ Sources and methods The objective of this section is to outline the data collection stage of the evaluation. ·■ Information should be provided on the balance between primary data generation and the use of secondary sources. This will allow evaluators to better plan the work programme and calculate its cost. Transversal evaluation of impacts on intervention priorities The development of the evaluation methodology will be undertaken in co­ operation with the Commission services and the relevant experts selected for the study. The evaluation methodology should comprise both qualitative and quantitative assessments. The study would involve two main elements: desk research and field work in the regions concerned. • Desk research The consultant should undertake a detailed comparative assessment of the areas for which he will be appointed. The review will concern past and current programmes (CSFs/SPDs, OPs and other) and their main compo­ nents relating to RTD activities such as regional analyses, strategic approaches, hierarchy of objectives, priorities and measures, indicators and profiles, synergy between RTD measures and other components. The study will draw extensively upon the contents of the programmes, and the available evaluation exercises for those programmes. In particular, it should take into consideration the methodology and lessons drawn from the RTD Thematic evaluation for Ireland, Portugal, Greece for the 1989-93 period. It should also take into account the results of recent evaluations such as the mid-term evaluations (for objective 1 regions) and the evaluation of RTPs (Regional Technology Plans) under the ERDF art. 10. The consultant is invited to collect data and statistics and to develop appro­ priate indicators which may be relevant for evaluation purposes in order to provide key elements on RTD expenditure, as well as the nature and the impacts of the actions funded. • Field work In order to derive first hand information on the effects of RTD related actions, the consultant will interview key informants in the regions concerned. A short list of key issues (to be carefully selected) should be used to structure the questionnaire or checklists for use in the interviews. A number of case studies will be selected to provide in-depth and contextual evidence and information on best practices on RTD related actions under the Structural Funds. Appendix Deliverables The deliverables (inception report, interim and final reports, case study reports, etc. ) that the evaluation will produce, both intermediate and final, should be listed and their contents detailed when necessary. • What format should be used for the production of documents? • What type of computer file should be used? • Which language(s) should be used? *■ The timetable for an evaluation is usually determined by the decision­ making calendar and not by methodological requirements. Rather than impose a timescale that may reduce the depth, scope or even quality of the evaluation, the production of an interim report can be an acceptable compromise. The proposed work programme and calendar should include: ­ a detailed work programme covering essential questions and a methodo­ logical framework for the thematic study ­ national reports covering the desk research and field work stages ­ a final synthesis report, including a summary focusing on policies to be followed, which must be understandable and usable independently of the main report. ­ All the texts and the corresponding database will be established or trans­ mitted (hard copy and diskette) in a format determined by DG XVI of the European Commission. ­ Reports and documents will be presented in English, French or German. A supplementary budget is planned for translation into at least one of the two languages. Calendar The duration of the evaluation should be indicated along with any inter­ mediate deadlines. The study, divided into two separate lots will be completed in one year. Transversal evaluation of impacts on intervention priorities Appendix 4: List of boxes Introduction Box 1 - Priorities, domains and themes Box 2 - Mainstreaming: a definition 8 11 Technical considerations relating to the assessment of thematic impacts Box 3 - Using objectives trees for clarifying implicit and explicit objectives related to a given theme Box 4 - In-depth analysis of impacts: appropriate tools and constructing an evaluation method Box 5 - Typical strengths and weaknesses of sources of existing information Box 6 - Estimating impacts net of deadweight Box 7 - Additionality, deadweight and leverage effects 21 23 25 20 16 Evaluating the environmental impacts of structural interventions Box 8 - An adaptation of the Driving Force-State-Response (DF-S-R) model of sustainable development Box 9 - Detailed typology of environmental impacts of structural interventions Box 10 - Environmental objectives of Structural Fund Interventions presented in the form of a thematic objectives tree. Box 11 - Example of a system of indicators for the construction of waste-water treatment plants Box 12 - Example of an environmental screening form 33 36 37 47 53 Evaluating the impact of structural interventions on the competitiveness of Small and Medium-Sized Enterprises Box 13 - Definition of SMEs Box 14- Example of a thematic objectives tree for SME interventions Box 15 - Example of a system of indicators for an intervention providing informational support for SMEs to promote exports Box 16 - Example of a factor analysis of the Catalan co-operative sector 59 65 71 74 Transversal evaluation of impacts on intervention priorities Box 17 - Example of the use of the case study technique in the evaluation of the NordTek programme, Denmark 75 Evaluating the impact of structural interventions on employment 80 80 Box 18 - Different types of skill obsolescence Box 19 - Breakdown of different types of employment effects Box 20 - Production of employment effects through structural interventions Box 21 - Structural Fund interventions and employment effect mechanisms Box 22 - Example of a thematic objectives tree for the promotion of employ­ ment Box 23 - Comparison of deadweight, displacement and substitution effects Box 24 - Example of a system of indicators for a return-to-work programme 86 89 97 82 82 Evaluating the impact of structural interventions on equal opportunities Box 25 - Example of a thematic objectives tree for the priority of equal opportunities Box 26 - Example of effects by category of impact Box 27 - Example of the application of the principle of mainstreaming Box 28 - Example of a system of indicators for the establishment and re-generation of community centres Box 29 - Standard form for assessing the equal opportunities dimension in the framework of ex ante evaluation Box 30 - Matrix for programme assessment 121 124 125 112 114 120 Evaluating impacts arising from interventions in the domain of Research and Technological Development Box 31 - Thematic evaluation of the impacts of CSFs for Research and Technology in Greece, Ireland and Portugal for the 1989 -1993 programming period Box 32 - Framework for the evaluation of Structural Fund interventions in the domain of RTD Box 33 - Example of a thematic objectives tree for RTD interventions 134 138 139 Appendix Box 34 - Example of a system of indicators for an intervention providing financial and informational support for technology acquisition by SMEs Box 35 - Example of a score card for a joint research venture 145 146 Evaluating impacts arising from interventions in the domains of transport, telecommunications and tourism Box 36 - Example of a system of indicators for upgrading of a railway line for freight operations 156 Box 37 - Example of a system of indicators for the deployment of a broadband Hybrid Fibre Coax (HFC) network Box 38 - Example of support for the creation of tourist facilities (museums, theme parks, etc. ) Box 39 - Example of a SWOT analysis matrix Box 40 - Extract from a "corridor ranking framework" applied to Greece 160 161 157 163 A step-by-step approach for thematic analysis Box 41 - Transversal evaluation of a theme : "Step by step" approach for a programme Box 42 - Thematic assessment matrix. Example relating to equal opportunities Box 43 - Score card. Example: assessing the extent to which a measure takes into account of the priority of equal opportunities Box 44 - Interviews, focus groups and case studies Box 45 - Summary table of measures and their effects. Example of improving the suitability of living conditions to women's needs Box 46 - Summary of the key steps and evaluation tools 170 173 174 175 176 177 Cross-programme thematic evaluation Box 47 - Comparison of supra-national synthesis of programme evaluations 180 and transnational evaluation 183 Box 48 - Sampling with the objective of highlighting best practice Box 49 - The development of transnational evaluations 184 Box 50 - Sampling with the objective of making a pan-European analysis Box 51 - Example of multiple-case studies with replication design Box 52 - The seven steps of research synthesis 185 187 188 Transversal evaluation of impacts on intervention priorities Index ethnographic observation, 123 Europe-wide analysis of impact, 179 evaluation research See research synthesis, 187 evaluation team, 16,120,169,178,181 evaluatory question, 17 expert panel, 148 external coherence, 15 externality, 137,148,164 -F- factor analysis, 74 feasibility study, 180 focus group, 73,123,175 framework conditions, 62 full-time employment, 94 full-time equivalent (FTE), 85 -G- gross employment impact, 23, 87 gross impact, 22 -I- impact descriptor, 172 implicit objective, 15 income multiplier effect, 24,25,82,88 indicator, 18 indicator (examples of), 47, 71,97,121,145, 156,157 indirect effect See induced effect, 155 induced effects, 155 information system, 20 input-output model, 104 internal coherence^ 5,38 interview technique, 73,123,127,149,160, 175 Ishikawa diagram, 123 job creation or maintenance effect, 77,80, 88,101 absorption, 86,133,137 activity rate, 89 ad hoc indicator, 44, 94,118 additional effects, 25 additionality, 25,137 attraction effect, 89,103 -B- baseline, 23,43,119 benchmarking, 182 best practice, 63,123,182 bibliometric indicator, 148 -C- case study technique, 54, 75101,123,125, 149,160,163,175,181 coefficients, 22,55,95,103,184 colour vote, 123 Community value-added, 179 comparative analysis, 102,123,186 confounding factor, 18,79,95 context indicator, 17,72,119,122,147,171 contingent valuation method, 55 control group, 87,102 cost-benefit analysis, 55,148,163 cost-effectiveness analysis, 101,135 counterfactual situation, 22,65,73, 87,99, 119 deadweight, 22, 38, 65, 87,101,103,140, 159 DELPHI technique, 164 demand-side effect, 155 displacement, 22,38,65,88,101,159 domain of intervention, 129 Driving Force-State-Response model, 33,43 -E- employability, 78, 79,95 employability effect, 80,88,101 employment effect, 25, 74,77,80 environmental impact assessment (EIA),29, 36,52 leverage effect, 24,66, 70,140,144 Transversal evaluation of impacts on intervention priorities -M- strategic environmental assessment (SEA), 32,50 substitution, 22, 65, 88,103,140,159 summary table, 176 supplier effect, 25,84,88 supply-side effect, 155 sustainable development, 29 sustainable employment, 84,94 SWOT analysis, 123,161,182 synergy effect, 65, 79,136,154,164 -T- technical committee, 180 temporary employment, 84, 93 terms of reference, 180 thematic assessment matrix, 172,177 transitory employment, 84 typology of impact, 35,113,139,172 -U- unemployment, 77,98,125 unemployment (effect on), 89 macro-economic model, 104 mainstreaming, 11, 70,107,120 matched pairs, 175 matrix of cross impacts (SMIC), 164 mitigation, 42 monitoring data, 73, 99,147,185 multi-country econometric analysis of impact, 184 multi-criteria analysis (MCA), 55 multiple case studies with replication design, 186 multiple case study technique, 55 -N- net employment impact, 79, 87 net impact, 22,38,66,101 -O- objectives tree, 16,37,65,86,112,139,171 -P- part-time employment, 94 policy-off situation, 22 political priority, 16,27 positive discrimination (affirmative action), 111 primary data, 20, 71,147 -Q- questionnaire survey, 73,101,123,127,148 reference situation, 17,96 regression analysis, 102 -s- sampling, 181,185 scenario technique, 123 scoping, 50, 52,171,174 score card, 146,173 scores and weights, 160 scoring See scores and weights screening, 50, 52,169 shift-share analysis, 102 spillover, 148 statistical surveys, 21,98 step-by-step approach, 169 Acknowledgements The preparation of this Volume took place over several years from 1996 to 1998. The pre­ paratory work included the following steps: Several workshops and meetings which allowed the points of view of experts in different themes and the transversal (or multi-thematic) perspective of those responsible for the Structural Funds to come together. This process firstly gave rise to the production of two MEANS Handbooks on employment (Handbooks 3 and 6) as well as a methodological note on evaluating human resource- related interventions. These documents were used to provide the basis of the chapter on employment. Similarly the chapters on the environment and equal opportunities are both based on recent methodological notes resulting from a confrontation of the ideas of experts and Commission officials. The consultation procedure employed during the preparation of the note on the environment is particularly noteworthy for its use of the Impact Mapping technique (see Volume 3 of the MEANS Collection). The production of this Volume was coordinated by Anastassios Bougas and David Sweet (D. G. XVI. G. 2 : Coordination of Evaluation) and involved several Directorate-Generals of the European Commission, notably D. G. II, V, VI, XI, XIV and XVI and in particular Philippe Goybet, Anne Bucher, Nelly Bandarra, Miguel Benito Alonso, R. Bradley, Ciaran Dearie, Sylvie Finne, Martin Hallet, Richard Harding, Mikael Kirosingh, Frédérique Lorenzi, Andrea Mairate, René Marty, Valéry Morard, Irmengard Nelles, Anastasios Nychas, Julius Op de Beke, Sergio Piccolo, Brigitta Renner-Loquenz, Jacqueline Soulier Oliveira, Martin Tyrrell, Martine Voets and Dominique Van Der Wee. This Volume has been written by the Centre for European Evaluation Expertise, notably with the participation, Cristina Diaz Gomez, Andreas Esche, Connor Kearney and Kevin Williams. It was translated into English by Liz Libbrecht. The authors wish to thank all those who contributed, either directly or indirectly, to the pro­ duction of this Volume, and in particular the following experts: Christine Abrossimov (DARES), Mariano d'Antonio (Universita da Roma), Peter Auer (Wissenschaftszentrum, Berlin), Jean-Claude Barbier (Centre d'Etudes pour l'Emploi), Martin Bekker (B&A Groep), Peter Bosch fThe European Environment Agency), Marco Causi (CLES Sri), Antonio Estevan Estevan (Gabinete de Economia Aplicada), Marta Ferrés (Interés General), Chris­ topher Hasluck (University of Warwick), Andy Hirst (PA Cambridge Economics Consul­ tants), Liisa Horelli (ITSU OY), Japp de Koning (Netherlands Economic Institute), Kierän A. Kennedy (The Economic and Social Research Institute), Ulla Koski (Finnish Ministry of Environment)Marie-Christine Lefebvre (Lconsult SPRLHedwig Lutz (WIFO), James Medhurst (Ecotec), Dr Heino Von Meyer. Chris Newton, (The Environmental Agency), John Rhodes (PA Cambridge Economics Consultants), Barry Sadler (Institute of Environmental Assessment), Claude Seibel (DARES), Thanos Skouras (Reseach Centre of the Athens University of Economics & Business), Antoni Soy (Centro d'Estudis de Planiflcacio), Joachim Spangenberg (Wuppertal Institute for Climate, Environment & Energy), Peter Tyler (PA Cambridge Economics Consultants), Pieter Van Run (CBEA), Claudia Villante (ISFOL). European Commission Evaluating socio-economic programmes Transversal evaluation of impacts on the environment, employment and other intervention priorities Luxembourg: Office for Official Publications of the European Communities, 1999 1999 — 215 p p. — 14. 8x21 cm ISBN 92-828-6626-2 Venta · Salg · Verkauf · Πωλήσεις · Sales · Vente · Vendita · Verkoop · Venda · Myynti · Försäljning BELGIQUE/BELGlE ÖSTERREICH Jean De Lannoy Avenue du Roi _02/Koninn__an 202 B­1190 Bruxe«es_russo[ Tel. (32-2)538 43 08 Fax (32-2) 538 06 41 E­mail, jean de. LannoyOintoooard. be URL: http www jean­ae­iannoy De La librairie européenne/ De Europese Boekhandel Rue de la Loi 244/Wotsttaal 244 B­1040 Pruid'es­Brussel Tel. (32-2)295 26 39 Fax (32-2) 735 08 60 E­mail: mailOlibeurop be URL: hrtpJ/www hbeurop. be Moniteur bel ge/Belg lac h S t u t s b l ad Rue de Louvam 40­42/leuvenseweg 40­42 B­1000 Bruxelles/­russel Tôt. (32­2)552 22 11 Fax (32­2) 511 01 84 DANMARK J. H. Schultz Information A/S Heretedvang 10­12 DK­2620 Albertslund Tit (45)43 63 23 00 Fax (45) 43 63 19 69 E­mail. schuttzOschuüz dk URL: htlp:/Avww Schultz dk DEUTSCHLAND Bundesanzeiger Vertag GmbH Veni iL'bsaDtei lung Amsterdamer Straße 192 D­50735 Köln Tel (49-221)97 66 80 Fax (49-221) 97 66 82 78 E­Mail: vertnebObundesanieiger. de URL: rittp^/www. bundesanzeiger. de ΕΛΛΑΔΑ/GREECE G. C. Eleftheroudakl· SA International Bookslore Panepisiimrøu 17 GR­10564 Athina Tel (30­1 ) 331 41 80/1/2/3/4/5 Fax (30­1)323 9821 E­mail: elebooksOnelof gr ESPAÑA Boletín Oficial del Estado Trafalgar. 27 E 28071 Madrid Tel (34)915 38 21 11 (Libros). 913 B4 17 15(Suscnp. ) Fax (34) 915 38 21 21 (Libros). 913 B4 17 14 (Suscflp. ) E­mail: clientes O com boe es URL httpi/www boe es Mundi Prensa U b r o s, SA Castello. 37 E­28001 Madrid Tel (34)914 36 37 00 Fax ¡34)915 75 39 98 E ­mail 11 br e na O mundiprensa. es URL: httpy/www. murtdipiensacom FRANCE Journal ottici«! Service des pubi­cali on s des CE 26. rue Desaix F­75727 Paris Cedex 15 Tel (33) 140 58 77 31 Fax (33) 140 58 77 00 URL: http //www journal'Officiel gouv Ir IRELAND Government Supplies Agency Publications Section 4­5 Harcourt Road Dublin 2 Tel (353­1)681 31 11 Fax (353­1)47527 60 ITALIA Licosa SpA Via Duca di Calabria. 1/1 Casella postale 552 1­50125 Firenze Tel (39) 055 64 83 1 Fax (39) 055 64 12 57 E­mail: hcosaOftbcc it URL: http //www. ftbcc. it/licosa LUXEMBOURG Messageries du livre SARL 5. me Raiffeisen L ­2411 Luxembourg Toi (352)4010 20 Fax (352) 49 06 61 E mail mailOmdl lu URL­ http //www. mdl lu NEDERLAND SOU Servicecentrum Uitgever* CiriMorfei Planton straat 2 Postbus 20014 2500 EA Den Haag Tel (31­70) 378 90 80 Fax (31­70) 378 97 83 E­mail sduOsdu n URL: utip. www sdu nl Manz'sche Verlags­ und Universitátsbuchhandlung GmbH Kohlmarkt 16 A­1014 Wien Tel. (43­1) 53 16 11 00 Fax (43­1)53 1611 67 E­Mail: bestellen β manz co at URL r, tip ://www. manz. at/index, htm PORTUGAL Distribuidora de Livros Bertrand L d. ' Grupo Benrand. SA Rua das Tertas dos Vales. ■'■ ­ A Apañado 60037 Ρ­2700 Amadora Tel. (351-1)495 90 50 Fax (351­1) 496 02 55 Imprensa NaclortaI­Casa da Moeda, EP Rua Marqués Sá da Bandeira, 16­A Ρ­1050 Lisboa Codex Tel. (351­1)353 03 99 Fax (351­1)353 02 94 E­mail del irtcmOmail lolepac pt URL: http^/www. incm. pl SUOMI/FINLAND Akateeminen Kirjakauppa/ Akademiska Bokhandeln Keskuskatu i /Cent raiga tan 1 PL/PB 128 FIN­00101 Helsinki/Helsingfors PJtfn (358­9) 121 44 18 V :. i. (358­9)121 44 35 Sähköposti. akatilausOakaloeminen. com URL. htlp^/inrww. akaieeminen com SVERIGE B T J AB Traktorvager 11 S­221 82 Lund TIn (46­46) 18 00 00 Fax (46­46) 30 79 47 E­post bljfìu­pubQblj. se URL: hnp://www b1j. se UNITED KINGDOM The Stationery Office Lid International Sales Agency 51 Nine Elms Lane London SW8 5DR Tel. (44­171)373 90 90 Fax (44­171)873 64 63 E­mail: ipa. enqumesOlheso. co uk URL http7/www the­stabonery­oftice co. uk ISLAND Boka bud La rusar Blondel Skólavordustig. 2 15­101 Reykjavik Tel (354) 551 56 50 Fax (354) 552 55 60 NORGE S wet s Norge AS O s le n j ovo ie n 18 Boks6512Et1erstad N - 0 6 0 6 O s_ Tel. (47­22) 97 45 00 Fax (47­22) 97 45 45 SCHWEIZ/SU ISSE/SVìZZERA Euro Info Center Schweiz _o OSEC Slamptenbachstrafle 85 PF 492 CH­8035Z_nch Tel. (41­1) 365 53 15 Fax (41­1) 365 54 II E­mail eicsOosecch URL: htlp://www osee ch/eics BAL GARDA Europrees Euromedia Lid 59. brvd Vilosha BG­1000 Solia Tel (359-2) 960 37 66 Fax (359-2) 980 42 30 E­mail MilenaCmboi cri bg CESKA REPUBLIKA ÚSIS NlS­prodejna H a velkova 22 CZ­l30 00Praha3 Tel (420­2)24 2314 86 Fax (420­2) 24 23 11 14 E­mail nkpospOdoc nis cz URL: http//ustscr cz CYPRUS Cyprus Chamber of Commerce and Industry PO Box 1455 CY­1509 Nicosia Tel (357-2)66 95 00 Fox (357­2) 66 10 44 Email demelrapOccci org cy HHVATSKA Medialrade Lid Pavia Hatza 1 HR­10000 Zagreb Tel. (385­1)481 94 1 Fax (385­1) 481 94 1 MAQYAflORSZAG Euro into Service Europa Hâz Margiiszigel PO Box 4 75 H­1396 Budapest 62 Tel- (36-1)350 80 25 Fax(36­t)350 90 32 E ­ m a 11 e υ r ο ι π ίο « î mail mata ν hu URL: http//www. eurotnlo. hiilnoex him MALTA Miller Distributors Ltd Malta International Airporl PO Box 25 Luqa LG A 05 Tel (356) 66 44 88 F a x ( 3 5 6 ) 67 67 99 E­mail: gwirthOusa. net POLSKA Ars Polona Krakowskie Przedmiescie 7 Skr. pocztowa 1001 PL­00­950 Warsiawa Tel (48­22)826 12 01 Fax (4S­22) 826 62 40 E­mail: ars pol ­ b e vy hsn com pi ROMANIA Euromedia Str. G­ral Berthelot Nr 41 RO­70749 Bucuresti Tel (40­1)3154403 Fax (40­11314 22 86 R05SIYA CCEC 60­letiya Oktyabrya Av 9 117312 Moscow Tel. (7-095) 135 52 27 Fax (7­095) 135 52 27 SLOVAKIA Centrum VTI SR Nam Slobody, 19 5K­61223 Bratislava Tel (421­7)54 41 83 64 Fax (421­7) 54 41 83 64 E­mail: europOlbb1. 5tlk. stuba. sk URL: http. '. 'www 5ltk. stuba sk 5 LOVEHIJA Gospodarski Vestnlk Dunajska cesia 5 SLO­10OO Ljublana Tel (386)613 09 16 40 Fax (386)61309 1645 E­mail: europe ovest m k si URL: httpJ/www. gveslnik. si TÙRKIYE Dünya Intotel AS 100. Yil Mahallessi 34440 TR­80O5O Bagolar­Istanbul Tel (90­212)629 46 89 Fax (90­212) 629 46 27 E­mail, intolel θ dunya­gazeie com. tr AUSTRALIA Hunter Publications PO Box 404 3067 Abbolsford. Victoria Tel (61­3)94 17 53 61 Fax (61­3) 94 1971 54 E­mail: jpda «es Ooze mail com au CANADA Les «dillons La Liberté Inc. 3020. chemin Samte­Foy G1X 3V Samle­Foy, Quebec Tel (1­418)658 37 63 Fax (1­800) 567 54 49 E­mail libertoOmodtom. oc ca Rr­nouf Publishing Co. Ltd 5369 Chemin Canotek Road Unii 1 K1J 9J3 Ottawa. Ontano Tel (1­613)745 26 65 Fax(1 ­613)745 76 60 E­mail: order deptOrenoutbooks. com URL: http //www renoufbooks com EGYPT The Middle East Observer 41 Shenf Street Cairo Tel (20-2)392 69 19 Fax (20­2) 393 97 32 E­mail matoudaOmeobserver com eg URL: htlp7/www. meobserver com. eg EESTI INDIA Eesti Ka u b i nd us­Toost vi skods (Estonian Chamber of Commerce and Industry) Toom­Kooli 17 E E­0001 Tallinn Tel (372)646 02 44 Fax(372)646 02 45 Email einloOkodaee URL: htlp. 'www koda ee EBIC India 3rd Floor. Υ Β Chavan Centre Gen J. Bhosale Marg 400 021 Mumbai Tel (91­22)282 6064 Fax ¡91­22)285 45 64 E­mail: BbicOglasbmOl vsnl. net. in URL: htlpy/www ebondia com ROY luteiiintiun:il 41. Mishmar Hayarden Street PO Box 13056 61130 Tel Aviv Tel. (972-3)649 94 69 Fax (972-3) 648 60 39 E­mail: royilOnetvision. net il URL: htlp^/www foyinl. co il Sub­agent for the Palestinian Authority: Index Information Services PO Box 19502 Jerusalem Tel. (972­2) 627 16 34 Fax (972­2) 627 12 19 Ρ Si­J apan Asahi Sanbancho Plaza «206 7­1 Sanbancho. Chi yoda­ku Tokyo 102 Tel (81­3)32 34 69 21 F_x(81­3Í32 34 6915 E­mail: books ü ps ι­japan co jp URL: httpiWwww psi ­japan com EBIC Malaysia Level 7. Wisma Hong Leong 1B Jalan Perak 50450 Kuala Lumpur Tel. (60­3) 262 62 98 Fax (60­3) 262 61 98 E­mail: ebic­klOmol. net. my Mundi Prensa Mexico. SA do CV Rio Panuco No 141 Colonia Cuauhtemoc MX­06500 Mexico. DF Tel (52­5) 533 56 58 Fax (52-5) 514 67 99 E­mail: 1015452361 Ocompuserve com EBIC Philippines 19th Floor, PS Bank Tower Sen Gil J. Puyal Avo. cor. Tindalo Si. Makati City Metro Manilla Tel (63-2)759 66 80 Fax (63-2) 759 66 90 E­mail: occpcomOglobe com ph URL: hltpJAwww eccp. com EBIC Sri Lanka Trans Asia Hotel 115 Sir chittampalam A Gardiner Mawatha Colombo 2 Tel. (94­1)074 71 50 78 Fax (94­1)44 87 79 E­mail: ebicsieitmin. com EBIC Thailand 29 Vanisse Building. Btrt Floor Soi Chidlom Ploenchl 10330 Bangkok Tel (66-2)655 06 27 Fax (66-2) 655 06 28 E­mail: ebicbkkOkscl5. thcom URL: httpJVvww obicbkk. org UNITED STATES OF AMERICA Berna n Associates 4611­F Assembly Dnve Lanham MD207Ó6 Tel (1­800) 274 44 47 (toll Iroe telephone) Fax (1­800) 865 34 50 (ton Iroe lax) E­mail: queryObeman. com URL http //www beman com Bille wenden Sla sich an ein Büro Ihrer Wahl/ Please contaci the sales office of your c h o i e ·/ Veuillez vous adresser BU bureau de venie de votre choix Office tor Official Publications ot lha European Communities 2. rue Mercier L­2985 Luxembourg Tel (352)29 29­42455 Fax (352) 29 29­42758 E­mail: into info Oopoce cec be URL: httpJ/eur­op. eu int The MEANS collection: "Evaluating socio-economic programmes" This six-volume set gives wider public access to the lessons of the MEANS programme which the European Commission introduced with a view to impro­ ving and promoting evaluation methods. Its publication meets a growing need for reliable benchmarks to deal with the problems of evaluation encoun­ tered when implementing public measures in general and Structural Fund ope­ rations in particular. In this context, the set constitutes a unique methodological guide that can solve the technical and organisational problems arising in the course of evaluation work and for which the existing literature provides little help. The collection is aimed not only at professional evaluators, for whom it will provide a valuable handbook, but also at non-specialists, in particular manag­ ing authorities and structural programme Monitoring Committees. Evaluation design and management Each volume is designed to be independent of the others, yet they are also complementary. Together, they deal with all aspects of the evaluation of socio­ economic programmes: I. II. Selection and use of indicators for monitoring and evaluation III. Principal evaluation techniques and tools IV. Technical solutions for evaluation within a partnership framework V. Transversal evaluation of impacts on the environment, employment and other intervention priorities VI. Glossary of 300 concepts and technical terms OFFICE FOR OFFICIAL PUBLICATIONS OF THE EUROPEAN COMMUNITIES _-2985 Luxembourg