Science of the Total Environment 628–629 (2018) 1542–1556 Contents lists available at ScienceDirect Science of the Total Environment j ourna l homepage: www.e lsev ie r .com/ locate /sc i totenvReviewHarvesting the promise of AOPs: An assessment and recommendationsAnnamaria Carusi a,⁎, Mark R. Davies b, Giovanni De Grandis c, Beate I. Escher d,e, Geoff Hodges f, Kenneth M.Y. Leung g, Maurice Whelan h, Catherine Willett i, Gerald T. Ankley j a Medical Humanities Sheffield, University of Sheffield, Medical School, Beech Hill Road, Sheffield S10 2RX, UK b QT Informatics Limited, Macclesfield SK10 5DS, UK c Science, Technology, Engineering and Public Policy (STEaPP), Boston House, 36-37 Fitzroy Square, London W1T 6EY, UK d UFZ – Helmholtz Centre for Environmental Research, 04318 Leipzig, Germany e Eberhard Karls University Tübingen, Environmental Toxicology, Centre for Applied Geosciences, 72074 Tübingen, Germany f Safety and Environmental Assurance Centre, Unilever, Colworth Science Park, Sharnbrook, Bedfordshire MK44 1LQ, UK g The Swire Institute of Marine Science and School of Biological Sciences, The University of Hong Kong, Pokfulam, Hong Kong, China h European Commission, Joint Research Centre (JRC), Ispra, Italy i The Humane Society of the United States, 700 Professional Drive, Gaithersburg, MD, 20879, USA j US Environmental Protection Agency, 6201 Congdon Blvd, Duluth, MN 55804, USAH I G H L I G H T S G R A P H I C A L A B S T R A C T• The AOP framework aims to increase efficiency of chemical safety assessments. • The stakeholder community for AOPs, however, is broader than chemical risk assessors. • There are scientific and social challenges to successfully engage all stakeholders. • Multi-faceted communication and governance strategies will address these challenges.⁎ Corresponding author. E-mail addresses: a.carusi@sheffield.ac.uk (A. Carusi), geoff.hodges@unilever.com (G. Hodges), kmyleung@hku.h (G.T. Ankley). https://doi.org/10.1016/j.scitotenv.2018.02.015 0048-9697/© 2018 The Authors. Published by Elsevier B.Va b s t r a c ta r t i c l e i n f oArticle history: Received 15 December 2017 Received in revised form 2 February 2018 Accepted 2 February 2018 Available online xxxx Editor: D. BarceloThe Adverse Outcome Pathway (AOP) concept is a knowledge assembly and communication tool to facilitate the transparent translation of mechanistic information into outcomes meaningful to the regulatory assessment of chemicals. The AOP framework and associated knowledgebases (KBs) have received significant attention and use in the regulatory toxicology community. However, it is increasingly apparent that the potential stakeholder community for the AOP concept and AOP KBs is broader than scientists and regulators directly involved in chemical safety assessment. In this paperwe identify and describe those stakeholderswho currently-or in the future- could benefit from the application of the AOP framework and knowledge to specific problems. We also summarize the challenges faced in implementing pathway-based approaches such as the AOP framework in biological sciences, and provide a series of recommendations to meet critical needs to ensure further progression of the framework as a useful, sustainable and dependable tool supporting assessments of both human health and the environment. Although the AOP concept has the potential to significantly impact the organization and interpretation of biological information in a variety of disciplines/applications, this promise can only be fully realizedKeywords: Toxicology Adverse outcomes pathways Risk assessment Environmental healthmark.davies@qtinformatics.com (M.R. Davies), g.degrandis@ucl.ac.uk (G. De Grandis), beate.escher@ufz.de (B.I. Escher), k (K.M.Y. Leung),maurice.whelan@ec.europa.eu (M.Whelan), kwillett@humanesociety.org (C.Willett), ankley.gerald@epa.gov . This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). 1543A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556through the active engagement of, and input from multiple stakeholders, requiring multi-pronged substantive long-term planning and strategies. © 2018 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http:// creativecommons.org/licenses/by/4.0/).Chemical management Social, ethical and legal aspectsContents1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1543 2. Features of the AOP framework and its AOP knowledge base . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1544 3. Stakeholders in the evolution of the AOP framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1546 3.1. Regulatory assessment of chemical risks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1546 3.2. Chemical safety assessment for industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1547 3.3. Product discovery and development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1547 3.4. Medicine and health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1547 3.5. Environmental quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1548 3.6. Academic applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1548 3.7. Non-governmental organizations (NGOs). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1549 4. Challenges to scaling up the AOP framework and its applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1549 4.1. Formal publication versus providing input to the AOP-KB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1549 4.2. Possible risks and burdens to stakeholders supporting the AOP framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1549 4.3. Quality assurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1549 4.4. Maintaining effective dialogue about the AOP framework among stakeholder groups . . . . . . . . . . . . . . . . . . . . . . . . . . 1550 4.5. Ethical, legal and social aspects associated with use of the AOP framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1550 4.6. Governance and sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1550 5. Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1551 5.1. Publishing, depositing and reviewing strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1551 5.2. Education and training: stakeholder-specific approaches and materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1551 5.3. Stakeholder-specific interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1551 5.4. Translation into application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1552 5.5. Governance and sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1553 5.6. Impact indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1553 6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1553 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1554 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15541. Introduction Legislative dictates and societal expectations require a different approach to chemical safety assessment than has been used in the past when extensive data were collected for only a handful of chemicals of concern either to humans or the environment. Currently there is a need to determine the potential biological effects of tens of thousands of existing and new substances in a cost-effective and timely manner, but resources for chemical testing currently are either static or decreasing. Hence, there is increasing reliance on approaches to efficiently generate data concerning the possible biological activity of chemicals through use of in silico models, in vitro assays, and short-term in vivo tests emphasizing molecular and biochemical measures of effects (National Research Council, 2007). This creates a challenge for effectively translating results from these types of mechanistic/pathwaybased analyses into information useful to those ultimately responsible for conducting chemical safety assessments. The Adverse Outcome Pathway (AOP) conceptual framework was developed to serve as a knowledge assembly and communication tool to facilitate the transparent translation of mechanistic data into outcomes meaningful to chemical safety assessment (Ankley et al., 2010). Specifically, the AOP framework facilitates the identification and evaluation of causal linkages across biological levels of organization, such that mechanistic responses can be reliably used in decision-making to protect from adverse effects, such as cancer in humans or suppression of reproduction of ecologically important species. Many regulatory agencies across the world have recognized the potential of AOPs in supporting more efficient assessments of chemical safety. One notable example involves the Organization for Economic Cooperation and Development(OECD). TheOECD initiated an international AOPdevelopment program in 2012 that has sponsored and contributed to a number of critical activities, including the publication of standardized approaches for AOP development and review, and the formation of an AOP-knowledgebase (AOP-KB) to support AOP development and dissemination (Villeneuve et al., 2014a, 2014b; Vinken et al., 2017). A critical motivation for OECD involvement was the recognition by member countries that a pathway-based approach to chemical safety can promote economic growth aswell as the highest levels of health and environmental protection (Perkins et al., 2015). As the AOP framework has evolved in the context of regulatory toxicology, it has become recognized that the potential stakeholder community is broader than those involved in chemical safety assessment, as illustrated in Fig. 1. For example, the frameworkmay complement activities to apply the tractability of mechanism-based approaches for addressing biomedical challenges (Benson, 2015; van Hasselt and van der Graaf, 2015; Visser et al., 2014), in safety evaluations associated with drug development (Mirams et al., 2014), systems toxicology (Stahl et al., 2015) and clinical trial simulations (Han et al., 2016). Our aimwith this paper is to present a strategic vision to help ensure that the AOP framework can realize its full potential for multiple stakeholder groups. The paper first details the unique features and qualities of the AOP approach and of the AOP framework, and goes on to profile a variety of stakeholders, describing how they stand to benefit from the further development of the AOP framework and its application to solving their problems. Our analysis includes consideration of the core obstacles to a full, long-termengagement of the stakeholder community with the framework. Finally, we conclude with a set of recommendations regarding the steps that need to be taken to address these Fig. 1. Stakeholders involved in the different areas of research and development in relation to AOP development and usage. 1544 A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556challenges, and to ensure development of the AOP framework as a sustainable tool for the application of our scientific knowledge of living systems in numerous decision-making contexts. This paper stems from a Pellston workshop held April 2–6, 2017 in Cornwall, ON, Canada. The workshop was coordinated by the Society of Environmental Toxicology and Chemistry (SETAC), and financially supported by multiple governmental and business organizations. Pellston workshops typically assemble 30–40 (invited) experts on a given topic in the environmental sciences, who are charged with defining and proposing paths forward to address challenge(s) in the topic area of interest. This Pellston workshop focused on a number of issues associated with AOP science and implementation, including the development of a "roadmap" to promote sustainable use of the concept. Specific needs addressed at the workshop were identified through a global horizon scanning effort that gathered input from scientists and risk assessors/managers from throughout the world (LaLone et al., 2017a). In all, 41 experts representing government, academia and business, from nine different countries in North America, Europe and Asia participated in the workshop. 2. Features of the AOP framework and its AOP knowledge base Over the past several decades the global scientific academic, pharmaceutical, chemical and personal care product communities, as well as government agencies, have invested billions of (US) dollars generating biological information. The goal has been both to extend basic scientific knowledge of how biological processes causally unfold and to inform the development of less harmful chemicals and more effective disease diagnosis, prevention and treatment. Traditionally, this information has been deposited in scientific publications, but the sheer number of papers, compounded by the fact that the information is often conveyed in unstructured and inconsistent ways, makes it difficult to access, share and integrate. Initiatives to summarize and compile this information into centralized data repositories have been only partially successful, as they i) do not include all publications, ii) remain compartmentalized, and iii) use different ontologies. These disconnected databases have resulted in the next level of "silo-ed" information. The rise in bioinformatics, including collections of "omics" information and associated relational databases, has also produced a vast amount of potentially useful, but still ultimately difficult-to-access information. An additional complication is that lack of curation makes it difficult to assess the quality of much information (Tripathi et al., 2016; Laegreid and Kuiper, 2015; Howe et al., 2008).The goal of the AOP framework is to compile and synthesize this wealth of biological information such that it can be transparently and efficiently employed for decision-making. Fig. 2 provides an overview of the AOP framework in the context of its potential application to the translation and use of different types of data to support assessment of the effects of chemicals on human health and the environment. The initial interaction of a chemical with a biological system is depicted as the molecular initiating event (MIE), such as binding to a protein (e.g., receptors, enzymes) or DNA, or interactions with membrane lipids. These MIEs can cause subsequent perturbations at higher biological levels of organization, depicted as intermediate key events (KEs) along an AOP, which ultimately may result in adverse apical responses such effects on survival, reproduction, carcinogenesis, etc. In the case of ecological assessments, impacts at the population level also often are an endpoint of regulatory concern. Implicit in an AOP is that the depicted KEs are causally-associated with one another via defined KE relationships, an attribute that can be assessed usingweight-of-evidence analyses (Becker et al., 2015). It is this documented, formalized linkage across biological levels of organization in anAOP that provides the basis for the use of "alternative" data streams to predict the types of apical responses deemed critical to regulations and risk assessments. The AOP framework potentially goes beyond biological pathways, but will improve our understanding of chemical toxicity in relation to the ecological impacts of chemical contaminants in air, water, soil and food as well as impacts to human health. Escher et al. (2017) give a detailed account of how the mechanistic understanding of the AOP framework can be useful beyond human health for broader ecological issues. The AOP framework has wide implications for advancing the assessment of chemical hazards across various environmental compartments covering atmosphere, hydrosphere, biosphere, lithosphere, and anthroposphere. The AOP-KB also has specifically been designed to address the problem of unstructured and scattered information. The AOP-KB currently consists of several software packages, all of which deal with collecting, formatting and, to some extent evaluating, biological information. At the time of writing, the AOP-Wiki is the most prominent and well-developed of the packages and therefore it is the illustrative focus in this paper; however, the needs and issues for engagementwith the other elements of the AOP-KB will likely be similar. The AOP-Wiki is an opensource platform for collecting and organizing biological information. At present the AOP-KB contains about 223 AOPs applicable to assessments focused on both human health and the environment (http:// aopkb.org, accessed 20 Sep 2017). Other AOP-KB packages include Fig. 2. Depiction of the role of the Adverse Outcome Pathway (AOP) framework in linking various data streams to outcomes relevant to regulatory decision-making for chemicals. MIE – molecular initiating event, KE – key event, KER – key event relationship, AO – adverse outcome. Source: From Ankley and Edwards (2018). 1545A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556Effectopedia, AOP Explorer, and the Intermediate Effects Data Base (Table 1). A unique feature of the AOP-KB is that each of the modules is being developed and managed by different agencies or entities, but the overall organization and supporting documents are centralized under the auspices of the OECD. It is not, however, the only database for AOPs, since there will soon be a standard reporting template that can be used to deposit AOPs in a network of interoperable databases or knowledge bases, with the AOP-KB as a central hub. Stakeholders need to be aware of all the unique features of the AOP framework that make it worthwhile for them to fully embrace it and motivate them to actively engage with the broader AOP community. At its core, the AOP framework is a comprehensive means of gathering, integrating, curating, sharing, reviewing and disseminating knowledge about the mechanisms and consequences of perturbation of normal biological function by chemical or non-chemical stressors in different organisms. Ultimately there are three "selling points" of the AOP framework, namely; i) the AOP knowledge itself, ii) the way the knowledge is assembled/treated and iii) the potential applications and impact the knowledge can have. The primary type of knowledge captured by the framework is obviously related to toxicological processes andmechanisms. However, in a description of an AOP there is also valuable information on test methods and models that can be used to measure or predict the KEs that comprise an AOP, and information on chemicalTable 1 Components of the AOP knowledge base (KB) (http://aopkb.org/, accessed 20 Sep 2017). Module Description Developing entity e.AOP.Portal Main entry point for the AOP-knowledge base (AOP-KB) OECD AOP Wiki An open-source platform for collecting and organizing biological information US Environ Office of Re Effectopedia An open-knowledge and structured platform able to display quantitative information on Adverse Outcome Pathways (AOPs) OECD AOP Explorer AOP network visualization and analysis tool US Army C Intermediate Effects Data Base Repository for key event information European C Centreinitiators and their relevant chemical properties linked to the MIE. Thus when one describes the type of knowledge covered by the AOP framework, it is important to highlight the availability of information on methods and chemicals associated with measurement of the KEs. Some of the key attributes of the way knowledge is managed within the AOP framework are depicted in Fig. 3(a). They include: Clarity – the description of an AOP is highly structured and follows a particular format and well defined conventions and guidance (Villeneuve et al., 2014a, 2014b; OECD, 2017, 2016a, 2016b, 2016c). Once users become familiar with the format, it provides clarity, ease and efficiency when accessing AOP knowledge and ultimately provides a de facto reporting standard. Synthesis – there is a wealth of existing knowledge relevant to perturbedbiological pathways spread acrossmany sources. AnAOP typically captures knowledge froma large number of peer-reviewedpapers and integrates and distills it into a concise form, resulting in considerable saving in time and effort for users. Sharing – the framework is built with crowdsourcing and knowledge-sharing very much in mind, not only for the initial development of an AOP but also during ongoing refinement and review. Since an AOP covers many levels of biological organization, this naturally stimulates and relies upon extensive collaboration across numerous scientific disciplinesURL http://aopkb.org/ mental Protection Agency search and Development http://www.oecd. org/chemicalsafety/testing/adverse-outcomepathways-molecular-screening-and-toxicogenomics. htm https://www.effectopedia.org/ orps of Engineers https://github.com/DataSciBurgoon/aopxiv ommission Joint Research aopkb.org Fig. 3. An illustration of (a) the attributes of the AOP framework and (b) the potential applications of AOPs. 1546 A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556Knowledgebase – themost important aspect of the framework is providing ready access to AOP content. Further, with the anticipated adoption of anOECDharmonised template for reporting AOPs and the recent launch of the e.AOP.Portal (Table 1), it is possible to envision interoperable AOP platforms maintained by different parties across the world. Transparency – AOP development and review processes are iterative in nature and full transparency helps to engage potential contributors directly, either as co-developers or reviewers, or indirectly through, for example, the posting of comments on the AOP-Wiki discussion pages. The identity of developers, contributors and reviews are also displayed within the AOP-KB. Quality – the quality of an AOP is assessed and assured through several steps: the availability of detailed guidance for developers (OECD, 2017, 2016a, 2016b, 2016c), continuous crowdsourced peer review, and curation by scientists involved in maintaining the AOP-KB. Moreover, AOPs developed within the context of OECD projects are subject to two rounds of rigorous expert review, one by an OECD expert advisory group, and the other by a panel of independent international experts. Endorsement – once an AOP has successfully passed through expert review within the OECD process, it can be submitted to the Working Party of National Coordinators for the Test Guidelines Programme and the Working Party for Hazard Assessment for their collective endorsement, leading ultimately to publication of the AOP in the recently created series dedicated to AOPs (OECD, 2016c). Quantification – the description of anAOP can be enhanced by the inclusion of quantitative data, such as the dynamics underpinning KE relationships (Villeneuve et al., 2014a; Wittwehr et al., 2017). In addition, more complex processes can be represented through the weighted combination of a set of interconnected AOPswithin a network (e.g., LaLone et al., 2017b). The AOP framework can be viewed by stakeholders from the perspective of its different potential applications (Fig. 3b). Some of these are quite closely related, such as reducing animal testing, predictive toxicology and safety assessment. Essentially, AOPs provide a mechanistic blueprint to design integrated assessment approaches based on combinations of in vitro and in silicomethods that predict toxicological effects of interest, thus reducing the need for animal testing (Tollefsen et al., 2014). Moreover, including species-specific knowledge within AOPs helps to address the increasing demand for species-relevant assessments. Mechanistic profiling of molecular libraries during the drug development process can be effectively informed by AOP knowledge to guide the selection of appropriate batteries of in vitro high-throughput screening (HTS) assays forfingerprinting the effects of chemicals to support their safety assessment. HTS bioassays can be employed to characterise not only single chemicals but also be used to test complex environmental samples in environmental monitoring (Schroeder et al., 2016). By using concepts of mixture toxicity and effect-directedanalysis, it is even possible to relate the effects observed in the complex environmental samples to known chemicals and to characterise the effects of mixtures of unknown chemicals (Brack et al., 2016; Neale et al., 2015). The AOP framework can also be described relative to current regulatory challenges and political concerns (Fig. 3b). Many agencies worldwide are under pressure to identify and regulate endocrine disruptors are looking to AOP-based approaches for hazard and risk assessment that can help fill data gaps within a reasonable timeframe, at an acceptable cost, and keeping animal testing to a minimum (Coady et al., 2017; Browne et al., 2017). The effects of "real-life" chemical mixtures represent another considerable challenge for risk assessors. In this context, AOP networks could prove to be a vital tool to explore mixture effects in order to focus attention on chemicals and AOPs that really matter (Villeneuve et al., 2014a, 2014b). Finally, the safe substitution of hazardous chemicals in products with less hazardous alternatives is an area of growing importance, but the use of traditional toxicity testing methods to inform the process are often too slow, ineffective and expensive. Thus, AOP-based hazard profiling and ranking potential substitutes based on their relative toxicity would be of great value tomany companies operating in a variety of sectors. 3. Stakeholders in the evolution of the AOP framework The AOP approach cannot deliver on its full promise without a critical mass of stakeholders, representing different disciplines and sectors, engaging it in different ways. There is a wide range of existing and potential stakeholders in the framework (Fig. 1). In this section, we profile seven areas and four groups of stakeholders who are already engaged (or could easily become more engaged) with the AOP framework. Each profile identifies a typical community of stakeholders, outlining why they are (or would be) interested in AOPs and how they could further develop the framework. The list is not exhaustive or definitive but ismeant to serve as an initial analysis of potential stakeholders and their roles for the advancement of the AOP framework. 3.1. Regulatory assessment of chemical risks A principal stakeholder group for the AOP framework and associated knowledge consists of regulatory toxicologists, risk assessors and managers directly involved in evaluating chemical safety and implementing risk mitigation measures. The responsibilities and needs within this group can be very diverse, depending upon jurisdiction, policy context and legislative mandate. Regulatory toxicologists and risk assessors have to deal with varied protection targets (e.g., various human groups or facets of the environment), different hazard/risk assessment scenarios, varying amounts of available data and/or levels of data collection capabilities (ranging from very little formost industrial chemicals, to large 1547A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556amounts for pesticides) and differing numbers of chemicals for which they are responsible (a few ranging up to thousands). Risk managers then have to use the output of a risk assessment to make decisions on whethermeasures should be taken as required by relevant laws and associated protection goals. Decisions might include restricting the use of a hazardous chemical or requiring its prior authorisation, and often include consideration of socio-economic consequences of intervention, which can be considerable. Accordingly, the utility and perception of value of the AOP framework will vary for different members of this stakeholder community. For example, assessors responsible for rapid processing of large numbers of chemicals with limited data may rely upon the AOP framework for the utilization of in silico and/or in vitro data for hazard profiling, which would serve as the basis for identifying chemicals of potential concern and prioritizing them for further testing or assessment. This scenario is exemplified by regulators involved in the US Environmental Protection Agency's (EPA)'s endocrine disruptor screening program, who use the AOP framework as a basis for prioritizing chemicals for possible in vivo testing based on their endocrine activity (Coady et al., 2017; Browne et al., 2017). Another scenario – such as in pesticide risk assessment – may involve members of a stakeholder group requesting the generation of new data, targeting tests/endpoints most likely to be sensitive to a given chemical. In this case, AOP knowledge can serve as a basis for guiding and optimizing test selection. 3.2. Chemical safety assessment for industry The chemical industry is required to provide information for specific apical endpoints relevant to human health and the environment to support safety assessments, as laid out under various regulations such as TSCA in the US (https://www.epa.gov/tsca-inventory, accessed 20 Sep 2017), REACH Regulation in the EU (European Parliament, 2006), and China REACH (Lau et al., 2012; Wang et al., 2012; CRS, 2014). These in turn are supported by guidance documents outlining the acceptability of methods concerning data requirements. These regulatory requirements are particularly relevant for companies involved in the development of new chemicals/products, but can vary according to industry sector (e.g., personal and homecare products, solvents, etc.). However, there are also a number of other safety requirements within industry that are driven by stewardship. These include identifying chemical alternatives that have reduced environmental or human health impacts (National Academy of Sciences, 2014), enhancing approaches for chemical read-across and categorisation (chemical grouping and extrapolation between chemicals), performing strategic/targeted testing (datagap filling), conducting biological read-across (interspecies extrapolation), identifying non-animal approaches to safety assessment, and conducting post-production environmental monitoring. Perhaps the greatest contribution that an AOP-based approach can provide to support regulatory toxicology and risk assessment is a reduction of the uncertainty in understanding hazardous effects for safety assessment (and hence increased confidence in the safety assessment) through a better understanding of pathway plausibility and biological read-across for targeted data-gap filling, etc. (Perkins et al., 2015). In the short term the unique AOP construct allows information on KEs and KE relationship-derived causal links from widely-varying data sources (ranging from in silico to in vivo) to enable supporting weight of evidence approaches to be used in, for example, biological readacross, without the need for apical endpoint testing (e.g., Burden et al., 2015). In the mid to longer term, the AOP approach has the potential to be used to help identify relevant pathways in the derivation of biological pathway altering doses (Judson et al., 2011) or for helping to define biological points of departure (Bercu et al., 2016), as part of the risk assessment paradigm shift outlined by the National Academy of Sciences in the US (NAS, 2014). One high-profile example of the use of AOPs in chemical safety assessment in this forward-looking manner has been in collating scientific data supporting an AOP for skin sensitization forthe purpose of assessing the risk of allergic reaction in humans without the necessity of animal testing (Maxwell et al., 2014; OECD, 2012; MacKay et al., 2013). 3.3. Product discovery and development The pharmaceutical (both human and veterinary) and agrochemical industries are primarily focussed on developing novel chemical entities for the purpose of medicinal and agricultural usage. Here a requirement is to efficiently identify compounds that may be harmful to human, animal health or the environment. The motivations are significant, as a number of chemicals used, for example, in commonly prescribed medicationsmay be hazardous to the environment (see reviewbyDaughton, 2003). A complementary need is the necessity to retain chemicals of proven therapeutic effectiveness. There are a number of opportunities to use the AOP framework to support these activities such as enriching the triaging process identifying undesirable chemical/pharmacological properties applied early in drug discovery to deprioritize chemicals likely to cause harm. Secondly, AOPs from theAOP-KB could act as a precursor for onward,more quantitative approaches based on systems toxicology (Sturla et al., 2014) for simulating the potential consequences on adverse outcomes that are still poorly characterised. For some types of adverse effects such as hepatotoxicity (Woodhead et al., 2017) and cardiac toxicity (Davies et al., 2016), there have already been demonstrations of using in silico approaches in thepharmaceutical industry. However, there is scope for extending consideration of these toxicities in support of preclinical safety assessment of drug candidates on sensorial systems, such as drug-induced disturbances on taste and smell or sleep disorders (Cavero and Holzgrefe, 2015; Cavero and Holzgrefe, 2017).Wider employment of the AOP framework to complement nascent approaches offers tangible benefits in being able to enrich the available chemistry for compounds that would be subsequently less likely to fail due to unforeseen toxicities or environmental harm. 3.4. Medicine and health The AOP framework would be enriched by the knowledge and expertise of clinicians, who could contribute understanding of the biological and physiological context of AOPs, and therefore of their plausibility or potential gaps. In turn, the AOP framework is applicable to understanding disease pathways across multiple biological levels and is a highly useful addition to the "toolbox" of approaches increasingly being used in the clinic for prevention, diagnosis and treatment, and in biomedical and clinical research for drug discovery, efficacy and safety testing (Langley et al., 2015; EFPIA MID3 Workgroup et al., 2016). Clinicians are important for fully developing the biomedical potential of the AOP framework, as they could increase access to humanbased data (Rodriguez et al., 2015), and give a more contextualised understanding of in vitro assays. The framework could also offer unique forms of support for clinicians using tools to ensure that prescribing is optimised for the patient. To this end, a number of decision support systems exist to help the clinician better tailor prescribed medicines that do not give rise to unintended adverse effects, such as co-medication error. Longer-term ambitions may include decision support systems that are tailored to individual patient characteristics beyond co-medications, such as diet, weight, sex and lifestyle. One such example is the CredibleMeds database (Schwartz andWoosley, 2016), which provides easy-to-use guidance to the clinician on the propensity for a particular medication to cause proarrhythmia, particularly when there are predisposing genetic factors. Whereas decision support systems in clinical contexts currently are largely based on statistical and epidemiological correlations, the AOP framework and AOP-KB would augment decision support by adding the layer of mechanistic data, making the biological basis for outcomes more transparent. This development would fit consistently with the current aspiration towards Personalised Medicine (e.g., pharmacogenomics), but would have to take heed of the 1548 A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556challenges that such attempts encounter at the level of clinical implementation (Hedgecoe, 2004; Day et al., 2017). The AOP framework also complements and supports the representation of disease progression, often referred to in medicine as the "natural history of disease". This might be particularly relevant to the identification of diagnosticmarkers of disease onset/progression,which could essentially correspond to discrete KEswithin anAOP. Future development would be particularly relevant for rare diseases, where progression is often poorly understood (FDA, 2015). Given the maturity of the AOPbased assessment for the likelihood of chemicals to cause skin sensitisation, this may substantiate the hope for further implementable system for clinician decision support. 3.5. Environmental quality Similar stakeholder groups as in chemical risk assessment are responsible for assessing the effects of chemical mixtures and complex environmental mixtures in the environmentin water and in food. Complex environmental mixtures pose a challenge that can be addressed more systematically through the application of AOP network concepts (Villeneuve et al., 2014a, 2014b). In vitro bioassays increasingly are being used for environmental monitoring to process many samples in a timeand cost-effective manner. Recently it has been suggested that in vitro HTS assays could provide water-monitoring programs focused either on human health (e.g., drinking water) or ecological (e.g., effluent discharge) effects (Schroeder et al., 2016). The main criteria for an in vitro assay to be applicable for monitoring complex environmental mixtures is that it sensitively detects chemicals of environmental concern, and that it results in a measurement that can be clearly related to relevant apical responses in organisms potentially exposed to the contaminant mixtures (Fig. 4). The AOP concept therefore can serve as justification for the choice of in vitro bioanalytical tools (Escher et al., 2014; Neale et al., 2017). By anchoring in vitro assay results (which typically reflect MIEs but in some cases also intermediate KEs) to relevant AOPs, this information is provided efficiently and, in turn, it is possible to develop new in vitro methods from knowledge of environmentally relevant AOPs (Fig. 4). 3.6. Academic applications Academic scientists currently involved with the AOP framework are mostly human health and environmental toxicologists, biochemists,Fig. 4. Application of in vitro bioassays for environmental quality sumolecular and cell biologists, pharmacologists and bioinformaticians. An example group of academics who have adopted the AOP concept are environmental toxicologists, who have pursued different facets of mechanistic research for many decades (Hermens et al., 1985; McKim et al., 1987; van Welie et al., 1992; van Wenzel and Opperhuizen, 1995; Escher and Hermens, 2002), but with the advent of the AOP concept were able to assemble this past work in the context of a unifying approach and common language. Academic researchers have dual incentives as stakeholders for the AOP framework: i) they have vocational motivations as scientists who aspire to advancing knowledge in their field and teaching students; and ii) they have career motivations. For the latter, publications are a main indicator of scientific productivity and performance, and necessary for career progression in terms of positions, resources, and promotions. While there is some variability depending on country and institutions, in general there is increasing pressure on academics to demonstrate the impact of their research beyond academia. If the importance of impact on policy-making and industrial innovation continue to grow, it could provide an important incentive to contribute to AOP development. Academic scientists are in a critical position to make a significant contribution to the uptake and establishment of the AOP framework through training new generations of researchers and graduates who will be employed in industry, policy and regulation. For example, academics are able to promote AOPs via course curricula and incorporating the AOP-KB and applications in toxicology and pharmacology courses. As exemplified in a teaching monograph by Escher and Leusch (2012) that heavily relies on theAOP framework, textbooks can greatly contribute to propagation of the AOP approach. An example of engagement of academic stakeholders with the AOP framework is the large European research consortium project SOLUTIONS (Brack et al., 2015), which has fully adopted AOPs as a unifying concept to address the challenges related to identifying and assessing water pollution. SOLUTIONS is an EU-funded research program with the goal of developing tools for the identification, prioritisation and assessment ofwater contaminants thatmay pose risks both to ecosystems and human health, to support the EU Water Framework Directive (Brack et al., 2017; European Parliament and European Council, 2000). SOLUTIONS relies heavily on the application of in vitro whole-organism and cell-based bioassays within an AOP framework for assessing the effects of complex mixtures of aquatic contaminants (Altenburger et al., 2015). Another prominent example of a major research consortiumrveillance monitoring that are anchored in the AOP framework. 1549A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556adopting the AOP framework, this time in the human health arena, is the EU Horizon2020 funded project, EU-ToxRisk (www.eu-toxrisk.eu). The project aims to employ non-animal methods to develop new approaches to assessing the safety of chemicals in a variety of sectors, addressing in particular their potential chronic, developmental and reproductive effects. 3.7. Non-governmental organizations (NGOs) NGOs give voice to concerns emerging from civil society and to the interests of those who are affected by chemical safety assessments and resulting decisions. Apart from this common goal, individual NGOs typically have differing, sometimes contrasting interests, and therefore have different incentives for engagement. Despite this, they are potential users of AOP knowledge and participants in developing and using the AOP framework. Engagement of the animal welfare NGOs stems from the desire to move away from the use of animals for chemical evaluation and biological research (Taylor et al., 2008). A companion concern is for improved human and environmental health, but the main focus is on the animals used in laboratories. Several animal welfare NGOs have recognized the promise of AOPs, and are participating in and supporting their development (see, for example, Sullivan et al., 2017) and promotion through education, training, and dissemination (see Table 3). These NGOs are also involved in promoting the AOP framework to the biomedical community (Langley et al., 2015, 2016). As environmental NGOs' core mission is to foster environmental protection, they are primarily concerned about the effects of chemicals on humans and the environment. There is a strong public concern about the large numbers of chemicals for which little safety information is available. They are also concerned about perceived and potential harm that is, or could be occurring via chemicals such as endocrine-active substances, and potential developmentally-toxic or neurotoxic chemicals (e.g., The Endocrine Disruption Exchange [TEDX], http:// endocrinedisruption.org/, last accessed 20 Sep 2017). Thus far, there has been only a somewhat limited engagement of environmental NGOs, with a few notable exceptions. Specifically, since pathwaybased information is used in research into the effects of pesticides on the environment (Kongsbak et al., 2014; Zhu et al., 2018) it can be expected that thiswill increasingly be used in the analyses of environmental NGOs, and potentially they could both participate in building AOPs, and benefit from the development and use of AOPs. At the same time, however, some environmental NGO groups have expressed concern regarding the scientific basis to the use of AOPs to support chemical safety decisions and have questioned the independence and transparency of the program at the OECD (Pesticide Action Network [PAN] EU, 2016). Increased participation of environmental NGOs could help confront these issues and facilitate their involvement in future developments. 4. Challenges to scaling up the AOP framework and its applications Broad utilization of the AOP framework is dependent on involving a criticalmass of stakeholders, whose participationwould ensure that the framework reflects and co-evolves with their current and future needs. There are a number of challenges in ensuring that the framework adequately engages and continues to meet the immediate and long-term needs of the different stakeholders profiled above; here we identify some of these challenges and in the subsequent section recommend cross-cutting strategies that can address different aspects of these challenges (see Table 2). 4.1. Formal publication versus providing input to the AOP-KB Themost significant hurdle for researchers for whom journal papers are an important output is the status of the AOP-KB relative to formal publication. Populating the AOP-KB currently requires considerablevoluntary effort, and the format is not necessarily compatible with the content of peer-reviewed articles that are the basis for developing impact metrics necessary for career advancement. Prior-publication rules imposed by most scientific journals would preclude publication if the information were already made available in the AOP-KB and, once published in a peer-reviewed journal, researchers may have little incentive to deposit their data in the AOP-KB. Potential copyright infringements also may occur through uploading previously published material onto the AOP-KB (such as the Wiki) (see Section 4.5 below). 4.2. Possible risks and burdens to stakeholders supporting the AOP framework While we take the view that the AOP framework is the most promising andmost realistic answer to the challenges of chemical safety testing, it is necessary to highlight possible risks and burdens, so as to address them. Policy-oriented organizations (like OECD) play a critical role in sustaining the credibility of the AOP framework and in providing assurance that AOPs can improve decision making. However, there are significant reputational risks, should there be non-optimal decision-making based on AOP information or knowledge. This risk increases as the popularity of the framework and the number of AOPs grow and the quality control process becomes itself increasingly crowdsourced and not limited to a tight community of highly dedicated scientists, as currently is the case. Governmental and industrial organizations involved in safety assessment are key players in securing the actual usefulness and hence the ultimate success of the AOP framework. But they are also those facing themost serious risks in adopting them, since they could be held accountable for harm to consumers, the environment or business interests. There needs to be confidence that decisions based on AOP considerations are as good as conventional approaches. Serious obstacles may come from lack of adequate incentives for developing and adopting new testing methods. For those in industry, the pressures to deliver assessments to an existing regulatory standard mean that there are few resources to consider alternative and new approaches.With significant initial development costs required to demonstrate robustness of a new approach, both in terms of intellectual and financial input, this means that often corporations are not in a position to support significant changes in an approach until it has already been sufficiently vetted by other organizations. Regulatory agencies are also resourceand time-strapped, such that adding another source of information to be weighed against other types of scientific evidence can make assessments more complicated. 4.3. Quality assurance AOPs can serve a number of functions ranging from, at their most basic, assembling available data as a basis for understanding what is and what is not known about a given pathway, to relatively complete, quantitative depictions of a pathway that could be applied to risk assessment. Accordingly, Villeneuve et al. (2014a) describe a spectrum of AOP development from "putative" to "quantitative". Becker et al. (2015) note that AOPs at all stages of this spectrum could be employed for different applications, but users need to consider the degree of their development and level of confidence in a given AOP in the context of whether it would be "fit for purpose" for a given application. Basically, the greater the impact of the application, the greater the confidence required in a supporting AOP. This requires that there be a formal process to assess AOP quality. From a science point of view, the issue of quality control is tightly interconnected with that of publication and peer review: the "gold standard" for quality assurance of research results. Accordingly, the AOP community has established a multi-level system of reviewing well-developed AOPs, internally and externally. For example, OECD-endorsed AOPs have a level of peer review equivalent or superior to that of a Table 2 Multiple means for addressing the different challenges. Publishing and review strategy Education and training Stakeholder specific interaction Translation into application Governance and funding structures Tensions between publication and knowledge base x x Risks: financial, reputational, etc. x x x Quality control x x Cross-disciplinary/sectoral understanding x x x Ethical, legal and social issues x x x Governance and sustainability x 1550 A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556published paper. OECDendorsement is recognized as bringingwith it an important reputational added value, especially by the regulatory community. Career wise for the academic community, there is still a preference for publishing in known, and high impact journals, even though they may recognise that the level of peer review is equivalent. The review process is also an important way of building the AOP community, as it draws in more contributors than those who develop and upload AOPs. However, there is a question whether such a work-intensive process can be sustained in the long term, especially in consideration of the lack of incentives for some of the reviewers. An increasing number of AOPs uploaded onto the AOP-KB would also make greater demands on the pool of qualified, often over-committed reviewers. 4.4. Maintaining effective dialogue about the AOP framework among stakeholder groups The AOP framework originally arose from the discipline of ecotoxicology to support regulatory decision-making (Ankley et al., 2010). This means that collaborators in other fields may not necessarily be interested in all its features, and may misunderstand aspects of the concept, or find the framework too constraining or simplified for their ownneeds. A recent survey described by LaLone et al. (2017a) identified a variety of concerns about AOPs, and attempted to address these using a "frequently asked questions" approach. For example, an issue often raised by researchers in biology, from academia and elsewhere, is the perceived linearity of the AOP, which is a misconception. Specifically, AOPs can be assembled into 'non-linear' networks which at times can be a better reflection of reality and thus be a focus of application (Villeneuve et al., 2014a, 2014b; LaLone et al., 2017b). It must be noted however, that although these provisions in AOP development allow for non-linarites and modulating factors to be included as narrative in the description of an AOP, the AOP construct itself was never intended to provide a means of representing a toxicological process as a formalized (mathematical) dynamical systems biology type model. An important challenge in stakeholders dialogue arises because of conflicting perceptions of the utility associated with different levels of detail included in the AOP-KB. The tension between development and communication in the AOP-KB is particularly prominent for different stakeholder communities reflecting "users" as opposed to "developers", with the former often requiring less complexity than the latter. 4.5. Ethical, legal and social aspects associated with use of the AOP framework There are legal issues around intellectual property rights (IPR) in the production and dissemination of AOPs that deserve attention and action. Since many scientists need first to publish the results of their research in academic journals, steps should be taken in order to enable scientists to reuse their work for contributing to or authoring AOPs without infringing on copyright ownership. Conversely, there is a need to enable researchers to reuse parts of their research that have been uploaded on the AOP-KB in publications without violating anti-plagiarism rules. In addition, AOPs published by theOECD do not specify what kind of copyright, if any, protects the intellectual work and which uses are authorised. This is strikingwhen intellectual work freely accessible through the Internet often comes with clearly specified licences. The same applies to what is available through the AOP Wiki, which by its nature as a wiki platform, is often associated with open access and free sharing or reuse of content. Authors need to have a clear perception of the kind of IPR protection available forwork shared through the AOPKB. AOPs are not only a tool for systematically organizing and curating scientific knowledge and indicating knowledge gaps that need to be filled, but have been promoted and are perceived as an important instrument for regulators dealing with protecting the environment and human health. As such AOPs may be used in regulatory contexts where the stakes and uncertainty are high, fierce controversies may arise and stakeholders may become polarized, as exemplified by recent activism around endocrine-disrupting chemicals in the EU (Le Monde, 2016). An obvious ethical issue that AOP usage brings to the fore is the widespread desire to reduce or ultimately replace animal testing for chemical safety assessment. Asnotedabove, this is a strongmotivator for animalwelfare NGO stakeholders, but it may also result in tensions with other ethical priorities, for example, associatedwithperceptions regarding the role of animal tests in the effective protection of humanhealth and the environment. The interplay and potential conflicts between these ethical priorities needs further analysis. The social aspects of AOPs are related to the fact that they can be used by many different parties with different agendas and purposes, and this could lead to misuses prejudicial for human or environmental safety, or economically inefficient and burdensome. Recent trends in science policy and research funding emphasise that scientists need to make an effort to foresee and prevent the misuse of their knowledge and results (Ziman, 2001; Guston and Sarewitz, 2002; von Schomberg, 2007, 2013; Stilgoe et al., 2013), as is also evident in the European Commission Horizon 2020 (n.d.)Science with and for Society programme (https://ec.europa.eu/programmes/horizon2020/en/h2020section/science-and-society, accessed 20 Sep 2017). A robust consensus about the appropriate conditions and domains of application of AOPs is therefore needed. In other words, different stakeholders need to agree on when (i.e., at what level of development) AOPs are fit for what purpose (e.g., prioritisation of chemicals for testing, to make a regulatory decision, or for use as evidence in courts etc.; Becker et al., 2015). In practice, this demands attention both to the production of knowledge (making sure that AOPs meet high standards of epistemic robustness and are not influenced by research susceptible to conflicts of interests or biases) and to the translation and integration of that knowledge into social processes. Such integration also raises the question of how to deal with incompleteness of scientific knowledge and whether and how to complement it with non-scientific knowledge dispersed in society (Funtowicz and Ravetz, 1993; Nowotny et al., 2001; De Grandis, 2016). 4.6. Governance and sustainability As the AOP framework continues to evolve, governance and coordination of the effort in the future would ideally involve all stakeholders 1551A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556asmuch as possible. Objectives of overall governance arguablywould be best achieved through an impartial organization that broadly represents the various stakeholder groups. This could be accomplished through a dedicated professional society charged with coordinating advances in AOP knowledge and communication for the common good of all. Currently, the Society for Advancement of AOPs (SAAOP; http://www. saaop.org/, accessed 20 Sep 2017) handles several practical logistics associated with the wiki module of the AOP-KB, including funding an open-access server, and providing support to ensure that information supporting new or existing AOPs is entered into the system in amanner consistent with established OECD guidelines. However, the SAAOP is comprised of a comparatively small group of people, none of whom are primarily dedicated to AOP-associated governance or coordination. This is not a sustainable long-term situation. Furthermore, to date, much of the investment in the basic AOP infrastructure (e.g., key software in the AOP-KB, training, educational and promotionalmaterials, etc.) has come via resources provided from a relatively limited number of stakeholder groups, principally the US EPA and the European Commission, often under the auspices of the OECD. Since there is no assurance of continuing availability of resources from these organizations, the development of a stable, long-term "business plan" for provision of resources to cover these direct costs including dedicated personnel needs to be defined. Additionally, there needs to be a formal vision to attract and involve other institutional investors. 5. Recommendations Many challenges that have been discussed in the previous section can be addressed in severalways. Solutions are proposed in this section; however, there is not always a one-to-onemapping between challenges and solutions. Instead, there are cross-cutting solutions to meet multiple challenges (Table 2). 5.1. Publishing, depositing and reviewing strategies The AOP framework is part of a rapidly changing publishing infrastructure that is affecting almost all sciences (Clark, 2014). For example, new data repository-oriented journals give rise to an opportunity to facilitate the process of integration between publication of AOPs in the AOP-KB and in peer reviewed journals. In addition, many journals already allow supplementary information to be deposited in accredited repositories, instead of as pdf files downloadable only from the journal's website. A possible step forward would be to negotiate with publishing houses the acceptance of the AOP-KB as a site for supplementary information for published papers, in a manner analogous to the approach used for genomic data archived in the NCBI GEO database (https:// www.ncbi.nlm.nih.gov/geo/, accessed 20 Sep 2017). For example, those involved in AOP governance could discuss the possibility of making deposits into the AOP-KB, or using the AOP framework standardized format for depositing in other databases mandatory with the publication of AOPs in journals such as Environmental Science and Technology, Environmental Toxicology and Chemistry and Toxicological Sciences, all of which publish AOP-oriented papers. An example of how this can be done is illustrated by Fay et al. (2017)whonote that "The examples presented in the following sections, ..., are available in the AOP knowledgebase ...; AOP and KE numbers noted below reflect entries within this database. For simplicity, KER numbers (direct and indirect) are not displayed in the AOP figures; however, the web addresses for each AOP and its corresponding KEs and KERs are provided in the Supplemental Data." There aremultiple benefits to be gained from this integrative model: the researchers get credit for their publications and deposited data, the AOP data are openly accessible to peer-reviewers, and the growth of the AOP-KB and related databases would be accelerated. Other possibilities in next generation publishing could include developing a database-only journal to complement the AOP-KB. There isan increasing trend towards database-only journals such as Molecular Ecology Resources (Wiley, http://onlinelibrary.wiley.com/journal/10. 1111/(ISSN)1755-0998, accessed 20 Sep 2017) and Scientific Data (Nature, https://www.nature.com/sdata/publish, accessed 20 Sep 2017) in which the authors publish their work by depositing the research data following a stringent format requirement, and provide detailed descriptions of methods used for data collection, quality control and assurance, processing and analyses. These database-only journals provide a new framework for data sharing, thereby accelerating the pace of scientific discovery. Such journals could be sponsored by technical societies whose membership includes large numbers of AOP stakeholders, such as the Society of Environmental Toxicology and Chemistry (SETAC) or the Society of Toxicology (SOT). Finally, it also might be possible to establish a journal whose main focus is AOPs, though this could be perceived as a relatively narrow field of science thus limiting the number of contributors and hence restricting its growth. Since the development of any new journal would take time and resources, and obtaining an ISI Impact Factor can take at least 3–5 years, this option could discourage academics from submitting to an AOP-only oriented journal during its early development phase. For any of these solutions, it is paramount that the data format be compatible and interoperable, so that information can be added onto the AOP-KBwith minimum effort. Steps to ensure this are being undertaken by the OECD. 5.2. Education and training: stakeholder-specific approaches and materials Outreach to various stakeholder groups requires different strategies and educational materials. Some common and overlapping characteristics of stakeholders that will affect communication strategies include whether they are AOP developers, users of AOP-based information, or simply interested in understanding the concepts and the science behind AOPs and their use. A list of existing guidance documents and training presentations is available from SAAOP.org, but other centralized repositories for this type of information are needed. To date, most outreach has focused on scientific training of current and potential AOP developers, and a number ofmaterials have been created for this purpose (Table 3) Developers of AOPs need to understand the history, background and potential uses of AOPs as well as information concerning the AOP-KB. AOP developers also need detailed guidance on collecting, organizing, naming and evaluating the data that inform an AOP. There is a need to create greater awareness of the trainingmaterial that is already available, and constant effort needs to be directed at creating suitable targeted material, and disseminating this to pertinent stakeholders. Finally, as university teaching evolves from learning content to mastering problem-solving tools, the AOP concept offers an opportunity for a structured approach to environmental toxicology and ecotoxicology. Curricula may include practical courses on the development of AOPs as team-building literature review courses or may apply the conceptual framework of logic and uniform structuring of information. New venues for spreading of AOP information that remain to be explored are the emerging massive open online course (MOOC) that range from free availability to formal inclusion into universities' curricula (Paton, 2014). At writing, the online training course AOP Wiki module is in the process of becoming a certification course, which could serve as the first version of an open online course. 5.3. Stakeholder-specific interaction Different stakeholder communities have different needs regarding how much information they require, how they prefer it presented, and how they most productively could interact with the AOP-KB and approach. Research shows that virtual collaborative environments require significant buy-in from the communities for Table 3 Examples of existing AOP training material. Source: (adapted from LaLone et al., 2017a). Source Content For developers OECD Information about AOP Approach: http://www.oecd. org/chemicalsafety/testing/adverse-outcome-pathways-molecular-screening-and-toxicogenomics.htm; Guidance documents: OECD, 2017, 2016a, 2016b, 2016c European Commission Joint Research Center Background information on AOPs and AOP Wiki: https://eurl-ecvam.jrc.ec.europa. eu/about-ecvam/networks-and-collaborations/adverse-outcome-pathways-aop Human Toxicology Project Consortium (HTPC) and the Physicians Committee for Responsible Medicine (PCRM) Introductory PowerPoint presentations and videos available at https://humantoxicologyproject. org/aops-101/ More in-depth video presentations about AOPs: https://humantoxicologyproject.org/about-pathways/ or through the AOP Learning Channel https://www.youtube.com/channel/UCeP-bYPc1CRD3Ieaqrl-ysg HTPC Online AOP training course consisting of two modules, an introduction to AOPs and a tutorial on using the AOP Wiki. Available for download from https://humantoxicologyproject. org/about-pathways-2/aop-online-course/, or run directly from https://aopwiki.org/ For scientists HTPC and PCRM Introductory PowerPoint presentations and videos available at https://humantoxicologyproject. org/aops-101/ More in-depth video presentations about AOPs: https://humantoxicologyproject.org/about-pathways/ or through the AOP Learning Channel https://www.youtube.com/channel/UCeP-bYPc1CRD3Ieaqrl-ysg HPTC AOP Online Course: https://humantoxicologyproject.org/about-pathways-2/aop-online-course/ or https://aopwiki.org/ OECD Effectopedia Channel https://www.youtube.com/channel/UCpAWpL0TS53rtPcwYvFC55Q PCRM 'Adverse Outcome Pathways: Path to Improved Chemical Tests without Animals' http://www.pcrm. org/AOPs SETAC http://setac.sclivelearningcenter.com/index.aspx?PID=9484&SID=215605 National Centre for the 3Rs (NCRs) Pathways Based Approaches Resource Page: https://www.nc3rs.org.uk/pathways-based-approaches-resource-page US Environmental Protection Agency Adverse Outcome Pathways Research Brief: https://www.epa.gov/chemical-research/adverse-outcome-pathway-aop-research-brief For non-scientists HTPC Toxicity 101 (text descriptions) (https://humantoxicologyproject.org/tox-101/) 1552 A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556which they are designed, but this depends on a number of factors, not least of which is the active participation of the communities in designing and developing the interface of the knowledgebase (Carusi and Reimer, 2010). The 'build it and theywill come' approach does not work in engaging stakeholder communities. We recommend participant-based approaches to usability studies and an iterative approach to platform development, as this addresses two issues at the same time: the usability of the interface for the different communities, and their investment in its success through their participation in developing it.5.4. Translation into application Although AOP development principles are well described in guidance documents (OECD, 2016a) and in the scientific literature (Villeneuve et al., 2014a, 2014b; Fay et al., 2017), the actual development of an AOP can be an extensive piece of work, equivalent to a systematic literature review. Case studies provide a very practical and powerfulmeans of demonstrating the overall approach to AOPdevelopment and are an ideal vehicle to engage and inform relevant parties, ranging from potential scientific contributors to decision-makers interested in application. Case studies aremost illustrative when the context in which the AOP is being developed is clearly described, together with explanation of the process undertaken to assemble and weigh the evidence (Fay et al., 2017; LaLone et al., 2017b; Horvat et al., 2017). Thus good, impactful case studies demonstrate the process asmuch as the results. To the extent possible, case studies would benefit from the activeinvolvement both of those developing and using AOPs/AOP KBs of interest. Case studies are also important to illustrate the application of AOPs in different chemical risk assessment contexts (ECHA, 2016) and are very complementary to general guidance (OECD, 2016b). For example, the 12 case studies published by the OECD concerning 'defined approaches' for determining the skin sensitisation potential of chemicals (OECD, 2016c) illustrate how the knowledge captured in one AOP (OECD, 2012) can provide themechanistic basis formany different solutions to the same problem. Moreover, these particular case studies contributed significantly to the decision by EU regulators in 2016 to change the standard information requirements under REACH (European Parliament, 2006), requesting that in the first instance chemicals be assessed in terms of their potential to trigger KEs of the skin sensitisation AOP instead of conducting the standard animal test. AOPs at different stages of development and backed by different levels of evidence are fit for different purposes (Becker et al., 2015). For instance, an incomplete AOP with important gaps may be very useful for a researcher interested in understanding which knowledge gaps need to be filled, but not appropriate for regulatory applications. As a tool facilitating the translation of scientific knowledge into regulatory decision making and promoting the production of the knowledge needed by regulators, the AOP-KB needs reflexivity in order to be responsive and responsible towards society. This means that the knowledge made available to several stakeholders through the AOP-KB should be presented so that each different group of stakeholders understands and interprets correctly its level of robustness and suitability for their own purposes. Reflexivity requires "being aware of the limits of 1553A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556knowledge and being mindful that a particular framing of an issue may not beuniversally held" (Stilgoe et al., 2013), and this demands that limitations recognized by the author of an AOP should be conveyed to potential users. It is important to work closely with stakeholders in developing ways of presenting AOPs that provide themwith the necessary information for using them appropriately. Considering that users are not yet fully well-versed in using the AOP-KB, that misuses could have a serious impact either on the environment or human health, and that misuse leading to serious harm may discredit a novel tool with great promise in the eye of the public, we suggest establishing a translational surveillance exercise: an active monitoring of decisions made on the basis of AOPs. The goal would be to promptly detect and flag up inappropriate uses that can have negative consequences for safety and bring the AOP framework into disrepute. Moreover, this would provide very useful feedback for improving the communication strategy of the AOP-KB. 5.5. Governance and sustainability Various alternatives are possible in terms of governance. While the role of the SAAOP could be expanded to support a broader stakeholder community, an alternative optionmight be to engage other professional societieswith a stake in AOPs. For example, the SETAC and SOT arewellestablished scientific organizations representing a diverse membership, many of whom already are stakeholders in the AOP community. There would be a variety of mutual benefits to establishing a governance and coordination body for AOP-oriented activities through a multipartite SETAC/SOT consortium of some type. To ensure sustainability and advancement of the AOP framework, adequate, ongoing financial support is a prerequisite for continuity and growth in the future. Currently, the AOP-KB (largely the AOPWiki) platform is supported by membership fees (from the SAAOP). While the OECD has funded the recent developments of Effectopedia and the e.AOP.Portal, the technical support and input for development of the AOP-Wiki are primarily contributed by volunteer experts. The current, very restricted, funding model will hinder the further growth of the AOP-KB, and even threatens its sustainability in its current form. Continuous funding would be required to expand the IT hardware (e.g., servers), and human resources, such as full-time personnel, to upgrade software, enrich bioinformatics analyses, maintain the online system and coordinate/organize promotion and outreach activities with a view to supporting further growth of the AOP platform. A critical role of a governance unit focused on AOPs would be development of a formal Business Plan, for example, a multi-year AOP Development Plan (ADP), presenting a sustainable financial budget and expense projections alongside with deliverables supporting the various stakeholder communities. Based on this ADP, a governance unit should be able to solicit resources from various potential funding agencies, charitable trusts and donors. Our vision is that this unit would be a non-profit NGO (consistent with the current SAAOP model), which in order to avoid potential conflicts of interest might impose a limitation on sources of funding. For instance, to have a governance body supported only by industry or depending too much on the contributions of one or a fewdonorswould be inappropriate and undermine independence. Consequently, it would be important that contributions supporting AOP development/systemmaintenance come from a variety of contributors from different sectors, and that none can exercise undue pressure on account of their relative financial weight. The GenBank (https://www.ncbi.nlm.nih.gov/genbank/, last accessed 20 Sep 2017), which is an open access, annotated database containing nucleotide sequences and their protein translations, could serve as a model for AOP governance/support. Initial funding from among others the National Institutes of Health, allowed for further collaboration with private enterprise. Early successes (Benton, 1990) cemented a lasting partnership between central government and bioinformatics industry in funding the bank.Following theGenBank example, an AOP governance body could initially seek resources from multiple sources representing the whole range of stakeholders and geographies. Since the AOP platform is an international project, the funding may come from various countries. We recommend that a nascent AOP Consortium consider development of partnerships with the United Nations Environment Program (UNEP) under their program for the Strategic Approach to International Chemicals Management (SAICM). SAICM aims to promote chemical safety around the world, and provides funding to support relevant projects. While funding is primarily directed to developing countries, SETAC's long term history of collaboration with SAICM could provide a good basis for exploring the possibility of getting support for the AOPKB in the context of the development of the "The Strategic Approach and sound management of chemicals and waste beyond 2020" (http://www.saicm.org/Meetings/FirstIntersessional/tabid/5463/ language/en-US/Default.aspx, accessed 20 Sep 2017). Funding agencies at country or regional levels alsomay be interested in supporting more targeted AOP endeavours. For example, the UKbased National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs) recently sponsored a call for proposals on the development of an AOP for cardiotoxicity under their strategic award funding scheme (https://www.nc3rs.org.uk/, accessed 20 Sep 2017). In the US, one could envision that various units within the National Institute of Health could include similar disease or pathway-specific calls, or include contribution of information to the AOP Wiki as a requirement for funding, in the sameway as the EU's Horizon 2020 programme has provided funding to the EU-ToxRisk consortium, (which includes a specific objective to develop AOPs for the KB). In addition to OECD, UN and other international organizations and private companies, support for the AOP framework might be sought from charitable foundations and trusts. Prominent foundations like the Bill andMelinda Gates Foundation, the Parker Foundation have funding programs whose remit could include AOPs development; likewise support may be sought from, for instance, the PEW Charitable Trust and J.P. Morgan. 5.6. Impact indicators With the desire to create and maintain momentum for the development, implementation and application of AOPs by stakeholders and to support a plan to facilitate these actions, there is a need to understand how successful the community has been at achieving its objectives. Here we recommend key impact indicators to measure success. Our criteria in selection of the key impact indicators are that they align to the objectives and should be simple to adopt and measure. If we define the overarching objectives of the strategy as i) to develop a sustainable platform for the derivation of AOPs, ii) to engage and encourage stakeholders in developing and using AOPs, and iii) to embed AOPs and AOP thinking into decision-making we propose some initial "Impact Indicators" to measure success (Table 4). Additional Impact Indicators may be adopted at later stages as the framework develops further. 6. Conclusion Toxicological knowledge and regulation are currently at a turning point; historical approaches to regulatory toxicology are no longer scientifically, economically or socially robust enough to dealwith the challenges of assessing chemicals that affect the environment, human and animal health. The AOP framework and associated knowledge has the potential to make a significant contribution to addressing these challenges, as well as parallel demands in biomedical fields. Realizing this potential is a two-pronged process: the first is to fully incorporate the AOP framework into toxicological research and related scientific fields; the second is to populate the AOP-KB with AOPs, making the constantly evolving knowledge on AOPs consistently organised and curated, accessible, shareable, and fit for purpose in regulatory and other decisionTable 4 Possible impact indicators to measure impact and success of the AOP framework and community initiatives. Success objective Impact indicator What Timeline 1 Knowledgebase (KB) metrics Increase in the number of Wiki and webpage hits Baselining within 6 months + repeated annually Increase in the number of AOPs developed and in the KB Baselining within 6 months + repeated annually Increase in the number of OECD peer reviewed AOPs Baselining within 6 months + repeated annually Increase in the number of active users of the KB Baselining within 6 months + repeated annually Clear sustainable governance and supported platform A defined governance structure and an understanding of funding sources to provide technical support Within 1 year 2 Increase in biological coverage Increase in pathways and species AOPs Baselining within 6 months + repeated annually Publications Increase in publications related to AOPs Baselining within 6 months + repeated biannually Citations Increase in citations of AOP papers Baselining within 6 months + repeated biannually Altmetrics Increase in including (but are not limited to) citations on Wikipedia, discussions on research blogs, mainstream media coverage, mentions on social networks such as Twitter. Baselining within 6 months + repeated annually Training and communications Increase in available online training materials and hits/uses of training materials. Baselining within 6 months and repeated biannually Uptake in awareness/acceptance and in formal training Survey of awareness and use of AOPs focussed initially on key stakeholders, SETAC/SOT members and establishments teaching toxicology courses (to include students). Baselining within 6 months and repeated in 2 years. 3 Uptake in decision making Assessment of the number in use in regulatory submissions (REACH (European Parliament, 2006) and PMN under TOSCA) and use in Investigational new drugs (IND) applications Assessment in 2 years of the number of applications including AOPs over the previous 2-year period. 1554 A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556making contexts. In this review, we have identified stakeholders who both stand to benefit and to make significant contributions to the AOP approach, discussed the main challenges to full involvement with the approach, and made far-reaching suggestions for a proactive, cohesive and targeted approach to fully realizing the potential of the AOP approach. Acknowledgments The authors thank the SETAC PellstonWorkshop co-chairs, C. LaLone and M. Hecker for their coordination, organization, and guidance of the workshop. We acknowledge the other workshop participants (see LaLone andHecker 2017) for their stimulating discussions and feedback and the respondents to the Horizon Scanning efforts for the charge questions and themes that informed our discussion. We appreciate the funding support from the Society of Environmental Toxicology and Chemistry, United States Environmental Protection Agency, American Cleaning Institute, Cefic-LRI (European Chemical Industry Council Long-Range Research Initiative), Chevron-Environmental, ECETOC (European Center for Ecotoxicology and Toxicology of Chemicals), European Commission Joint Research Centre, European Crop Protection, Wellcome Trust, ExxonMobil, Humane Society International, The Humane Society of the United States, Human Toxicology Project Consortium, Syngenta, and Unilever. In addition, we thank the groups from academia, industry, and government who supported participants' travel. Research conducted by Annamaria Carusi and Giovanni De Grandis was funded by Wellcome Trust Grant Number 201576/Z/16/Z. The views and statements expressed in this paper are those of the authors alone and do not necessarily represent the views of the organizations to which the authors are affiliated. Thus those organizations cannot accept any responsibility for the views or statements expressed by the authors. References Altenburger, R., Ait-Aissa, S., Antczak, P., Backhaus, T., Barcelo, D., Seiler, T.B., et al., 2015. Future water quality monitoring adapting tools to deal with mixtures of pollutants in water resource management. Sci. Total Environ. 512-513, 540–551.Ankley, G.T., Edwards, S.W., 2018. The adverse outcome pathway: a multifaceted framework supporting 21st century toxicology. Curr. Opin. Toxicol. (Accepted). Ankley, G.T., Bennett, R.S., Erickson, R.J., Hoff, D.J., Hornung, M.W., Johnson, R.D., et al., 2010. Adverse outcome pathways: a conceptual framework to support ecotoxicology research and risk assessment. Environ. Toxicol. Chem. 29, 730–741. Becker, R.A., Ankley, G.T., Edwards, S.W., Kennedy, S.W., Linkov, I., Meek, B., et al., 2015. Increasing scientific confidence in adverse outcome pathways: application of tailored Bradford-Hill considerations for evaluating weight-of-evidence. Regul. Toxicol. Pharmacol. 72, 514–537. Benson, N., 2015. Network-based discovery through mechanistic systems biology. Implications for applications-SMEs and drug discovery: where the action is. Drug Discov. Today Technol. 15:41–48. https://doi.org/10.1016/j.ddtec.2015.07.003 (Epub 2015 Sep 10. Review. PubMed PMID: 26464089). Benton, D., 1990. Recent changes in the GenBank on-line service. Nucleic Acids Res. 18, 1517–1520. Bercu, J.P., Morinello, E.J., Sehner, C., Shipp, B.K., Weideman, P.A., 2016. Point of departure (PoD) selection for the derivation of acceptable daily exposures (ADEs) for active pharmaceutical ingredients (APIs). Regul. Toxicol. Pharmacol. 79, S48–S56. Brack, W., Altenburger, R., Schueuermann, G., Krauss, M., Herraez, D.L., van Gils, J., et al., 2015. The SOLUTIONS project: challenges and responses for present and future emerging pollutants in land and water resources management. Sci. Total Environ. 503, 22–31. Brack, W., Ait-Aissa, S., Burgess, R.M., Busch, W., Creusot, N., Di Paolo, C., Escher, B.I., Hewitt, L.M., Hilscherova, K., Hollender, J., Hollert, H., Jonker, W., Kool, J., Lamoree, M., Muschket, M., Neumann, S., Rostkowski, P., Ruttkies, C., Schollee, J., Schymanski, E.L., Schulze, T., Seiler, T.-B., Tindall, A.J., Umbuzeiro, G.D.A., Vrana, B., Krauss, M., 2016. Effect-directed analysis supporting monitoring of aquatic environments an in-depth overview. Sci. Total Environ. 544, 1073–1118. Brack, W., Dulio, V., Agerstrand, M., Allan, I., Altenburger, R., Brinkmann, M., et al., 2017. Towards the review of the European Union Water Framework Directive: recommendations for more efficient assessment and management of chemical contamination in European surface water resources. Sci. Total Environ. 576, 720–737. Browne, P.D., Noyes, P.M., Casey, W.M., Dix, D.J., 2017. Application of adverse outcome pathways to US EPA's endocrine disruptor screening program. Environ. Health Perspect. https://doi.org/10.1289/EHP1304. Burden, N., Sewell, F., Andersen, M.E., Boobis, A., Chipman, J.K., Cronin, M.T.D., et al., 2015. Adverse outcome pathways can drive non-animal approaches for safety assessment. J. Appl. Toxicol. 35, 971–975. Carusi, A., Reimer, T., 2010. Virtual Research Environment Collaborative Landscape Study. JISC, Bristol http://citeseerx.ist.psu.edu/viewdoc/download?doi= 10.1.1.404.6517&rep=rep1&type=pdf, Accessed date: 20 September 2017. Cavero, I., Holzgrefe, H., 2015. 14th Annual Meeting of the safety Pharmacology Society: threading through peripheral and central nervous system presentations. J. Pharmacol. Toxicol. Methods 75, 143–157. Cavero, I., Holzgrefe, H., 2017. 15(th) AnnualMeeting of the Safety Pharmacology Society: focus on traditional sensory systems. J. Pharmacol. Toxicol. Methods 83:55–71. https://doi.org/ 10.1016/j.vascn.2016.09.003 (Epub 2016 Sep 19. PubMed PMID: 27659846). Clark, T., 2014. Next Generation Publishing and the Web of Data. Semantic Web. 5/4 pp. 257–259. 1555A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556Coady, K.K., Biever, R.C., Denslow, N.D., Gross, M., Guiney, P.D., Holbech, H., et al., 2017. Current limitations and recommendations to improve testing for environmental assessment of endocrine active chemicals. Integr. Environ. Assess. Manag. 13, 302–316. CRS, 2014. New Chemical Notification Process in China – China REACH. http://www.cirsreach.com/China_Chemical_Regulation/IECSC_China_REACH_China_New_Chemical_ Registration.html, Accessed date: 20 September 2017. Daughton, C.G., 2003. Cradle-to-cradle stewardship of drugs for minimizing their environmental disposition while promoting human health. II. Drug disposal, waste reduction, and future directions. Environ. Health Perspect. 111, 775–785. Davies, M.R., Wang, K., Mirams, G.R., Caruso, A., Noble, D., Walz, A., et al., 2016. Recent developments in using mechanistic cardiac modelling for drug safety evaluation. Drug Discov. Today 21:924–938. https://doi.org/10.1016/j.drudis.2016.02.003 (Epub 2016 Feb 15. Review. PubMed PMID: 26891981; PubMed Central PMCID: PMC4909717). Day, S., Coombes, R.C., McGrath-Lone, L., Schoenborn, C., Ward, H., 2017. Stratified, precision or personalised medicine? Cancer services in the 'real world' of a London hospital. Sociol. Health Illn. 39, 143–158. De Grandis, G., 2016. Practical integration: the art of balancing values, institutions and knowledge–lessons from the history of British public health and town planning. Stud. Hist. Philos. Sci. C: Stud. Hist. Philos. Biol. Biomed. Sci. 56, 92–105. EFPIA MID3 Workgroup, Marshall, S., Burghaus, R., Cosson, V., Cheung, S., Chenel, M., DellaPasqua, O., Frey, N., Hamrén, B., Harnisch, L., Ivanow, F., Kerbusch, T., Lippert, J., Milligan, P., Rohou, S., Staab, A., Steimer, J., Tornøe, C., Visser, S., 2016. Good practices in model-informed drug discovery and development: practice, application, and documentation. CPT Pharmacometrics Syst. Pharmacol. 5, 93–122. Escher, B.I., Hermens, J.L.M., 2002. Modes of action in ecotoxicology: their role in body burdens, species sensitivity, QSARs, and mixture effects. Environ. Sci. Technol. 36, 4201–4217. Escher, B., Leusch, F., 2012. Bioanalytical Tools inWater Quality Assessment. IWA Publishing, London, UK. Escher, B.I., Allinson, M., Altenburger, R., Bain, P., Balaguer, P., Busch, W., Crago, J., et al., 2014. Benchmarking organic micropollutants in wastewater, recycled water and drinking water with in vitro bioassays. Environ. Sci. Technol. 48, 1940–1956. Escher, B.I., Hackermüller, J., Polte, T., Scholz, S., Aigner, A., Altenburger, R., Böhme, A., Bopp, S.K., Brack, W., Busch, W., Chadeau-Hyam, M., Covaci, A., Eisenträger, A., Galligan, J.J., Garcia-Reyero, N., Hartung, T., Hein, M., Herberth, G., Jahnke, A., Kleinjans, J., Klüver, N., Krauss, M., Lamoree, M., Lehmann, I., Luckenbach, T., Miller, G.W., Müller, A., Phillips, D.H., Reemtsma, T., Rolle-Kampczyk, U., Schüürmann, G., Schwikowski, B., Tan, Y.-M., Trump, S., Walter-Rohde, S., Wambaugh, J.F., 2017. From the exposome to mechanistic understanding of chemical-induced adverse effects. Environ. Int. 99, 97–106. European Chemicals Agency (ECHA), 2016. New Approach Methodologies in Regulatory Science Proceedings of a Scientific Workshop. Helsinki, April 2016. https://doi. org/10.2823/543644. European Commission Horizon 2020 ScienceWith and for Society. Available at: https:// ec.europa.eu/programmes/horizon2020/en/h2020-section/science-and-society, (accessed 20 Sep 2017) no date. European Parliament and European Council, 2000. Directive 2000/60/EC of the European Parliament and the Council of 23 October 2000 establishing a framework for Community action in the field of water policy (short: Water Framework Directive). Off. J. Eur. Communities 327/1. European Parliament and the Council of the European Union, 2006. REACH Regulation (EC) No 1907/2006 of the European Parliament and of the Council. http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:02006R1907-20140410&from=EN. Fay, K.A., Villeneuve, D.L., LaLone, C.A., Song, Y., Tollefsen, K.E., Ankley, G.T., 2017. Practical approaches to adverse outcome pathway (AOP) development and weight-of-evidence evaluation as illustrated by ecotoxicological case studies. Environ. Toxicol. Chem. 36, 1429–1449. FDA, 2015. Disease Natural History Database Development. https://www.federalregister. gov/documents/2015/08/17/2015-20130/disease-natural-history-database-development-u24 (accessed 20 Sep 2017). Funtowicz, S.O., Ravetz, J.R., 1993. Science for the post-normal age. Futures 25, 739–755. Guston, D.H., Sarewitz, D., 2002. Real-time technology assessment. Technol. Soc. 24, 93–109. Han, K., Claret, L., Piao, Y., Hegde, P., Joshi, A., Powell, J.R., et al., 2016. Simulations to predict clinical trial outcome of bevacizumab plus chemotherapy vs. chemotherapy alone in patients with first-line gastric cancer and elevated plasma VEGF-A. CPT Pharmacometrics Syst Pharmacol. 5, 352–358. van Hasselt, J.G., van der Graaf, P.H., 2015. Towards integrative systems pharmacology models in oncology drug development. Drug Discov. Today Technol. 15, 1–8. Hedgecoe, A., 2004. The Politics of PersonalisedMedicine: Pharmacogenetics in the Clinic. Cambridge. Cambridge University Press, UK. Hermens, J., Könemann, H., Leeuwangh, P., Musch, A., 1985. Quantitative structure-activity relationships in aquatic toxicity studies of chemicals and complex mixtures of chemicals. Environ. Toxicol. Chem. 4, 273–279. Horvat, T., Landesmann, B., Lostia, A., 2017. Adverse outcome pathway development from protein alkylation to liver fibrosis. Arch. Toxicol. 91:1523. https://doi.org/10.1007/ s00204-016-1814-8. Howe, D., Costanzo, M., Fey, P., Gojobori, T., Hannick, L., Hide, W., et al., 2008. The future of biocuration. Nature 454, 47–50. Judson, R.S., Kavlock, R.J., Setzer, R.W., Cohen Hubal, E.A., Martin, M.T., Knudsen, T.B., et al., 2011. Estimating toxicity-related biological pathway altering doses for high-throughput chemical risk assessment. Chem. Res. Toxicol. 24, 451–462. Kongsbak, K., Vinggaard, A.M., Hadrup, N., Audouze, K., 2014. A computational approach to mechanistic and predictive toxicology of pesticides. ALTEX 31 (1):11–22. https:// doi.org/10.14573/altex.1304241 (2014).Laegreid, A., Kuiper, M., 2015. Health and the information commons. Pan Eur. Netw. Gov. 13, 144–145. LaLone, C.A., Ankley, G.T., Belanger, S.E., Embry, M.R., Hodges, G., Knapen, D., et al., 2017a. Advancing the adverse outcome pathway framework-an international horizon scanning approach. Environ. Toxicol. Chem. 36, 1411–1421. LaLone, C.A., Villeneuve, D.L., Wu-Smart, J., Milsk, R.Y., Sappington, K., Garber, K.V., et al., 2017b. Weight of evidence evaluation of a network of adverse outcome pathways linking activation of the nicotinic acetylcholine receptor in honey bees to colony death. Sci. Total Environ. 584–585, 751–775. Langley, G., Austin, C.P., Balapure, A.K., Birnbaum, L.S., Bucher, J.R., Fentem, J., et al., 2015. Lessons from toxicology: developing a 21st-century paradigm for medical research. Environ. Health Perspect. 123, 268–272. Langley, G., Adcock, I., Busquet, F., Crofton, K., Csernok, E., Giese, C., et al., 2016. Towards a 21st century roadmap for biomedical research and drug discovery: consensus report and recommendations. Drug Discov. Today 22, 327–339. Lau, M.H.Y., Leung, K.M.Y., Wong, S.W.Y., Wang, H., Yan, Z.G., 2012. Environmental policy, legislation and management of persistent organic pollutants (POPs) in China. Environ. Pollut. 165, 182–192. Le Monde, 2016. Let's Stop the Manipulation of Science. http://www.lemonde.fr/idees/article/2016/11/29/let-s-stop-the-manipulation-of-science_5039867_3232.html. MacKay, C., Davies, M., Summerfield, V., Maxwell, G., 2013. From pathways to people: applying the adverse outcome pathway (AOP) for skin sensitization to risk assessment. ALTEX 30, 473–486. Maxwell, G., MacKay, C., Cubberley, R., Davies, M., Gellatly, N., Glavin, S., et al., 2014. Applying the skin sensitisation adverse outcome pathway (AOP) to quantitative risk assessment. Toxicol. in Vitro 28, 8–12. McKim, J.M., Bradbury, S.P., Niemi, G.J., 1987. Fish acute toxicity syndromes and their use in the QSAR approach to hazard assessment. Environ. Health Perspect. 71, 171–186. Mirams, G.R., Davies, M.R., Brough, S.J., Bridgland-Taylor, M.H., Cui, Y., Gavaghan, D.J., et al., 2014. Prediction of Thorough QT study results using action potential simulations based on ion channel screens. J. Pharmacol. Toxicol. Methods 70, 246–254. National Academy of Sciences, 2014. A Framework to Guide Selection of Chemical Alternatives. National Academies Press, Washington DC. National Research Council (NRC), 2007. Toxicity Testing in the 21st Century: A Vision and a Strategy. National Academy Press, Washington, DC. Neale, P.A., Ait-Aissa, S., Brack,W., Creusot, N., Denison, M.S., Deutschmann, B., et al., 2015. Linking in vitro effects and detected organic micropollutants in surface water using mixture-toxicity modeling. Environ. Sci. Technol. 49, 14614–14624. Neale, P.N., Altenburger, R., Ait-Aissa, S., Brion, F., Busch, W., de Aragão Umbuzeiro, G., et al., 2017. Development of a bioanalytical test battery for water quality monitoring: fingerprinting identified micropollutants and their contribution to effects in surface water. Water Res. 123, 734–750. Nowotny, H., Scott, P., Gibbon, M., 2001. Re-Thinking Science. Knowledge and the Public in an Age of Uncertainty. Polity Press, Cambridge UK and Maiden MA. OECD, Organisation for Economic Co-operation and Development, 2016a. Users' Handbook Supplement to the Guidance Document for Developing and Assessing Adverse Outcome Pathways. OECD Publishing, Paris FR https://doi.org/10.1787/ 5jlv1m9d1g32-en, Accessed date: 4 May 2017. OECD, Organisation for Economic Co-operation and Development, 2016b. Guidance Document for the Use of Adverse Outcome Pathways in Developing Integrated Approaches to Testing and Assessment (IATA). OECD Publishing, Paris FR http://www. oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=env/jm/mono(2016) 67&doclanguage=en. OECD, Organisation for Economic Co-operation and Development, 2016c. Guidance Document on the Reporting of Defined Approaches and Individual Information Sources to be UsedWithin Integrated Approaches to Testing and Assessment for Skin Sensitisation. OECD Publishing, Paris FR https://doi.org/10.1787/20777876. OECD, Organization for Economic Co-operation and Development, 2012. The Adverse Outcome Pathway for Skin Sensitisation Initiated By Covalent Binding to Proteins. OECD Publishing, Paris. OECD, Organization for Economic Co-operation and Development, 2017. Guidance Document on Developing and Assessing Adverse Outcome Pathways. OECD Publishing, Paris FR. PAN EU, Pesticide Action Network Europe, 2016. AOP, the Trojan Horse for Industry Lobby Tools? http://www.pan-europe.info/sites/pan-europe.info/files/public/resources/reports/pan-europe-aop-report-8.8-dec-16.pdf, Accessed date: 4 June 2017 Paton, C., 2014. Massive open online course for health informatics education. Healthc. Inform. Res. 20 (2):81–87. https://doi.org/10.4258/hir.2014.20.2.81. Perkins, E., Garcia-Reyero, N., Edwards, S., Wittwehr, C., Villeneuve, D., Lyons, D., Ankley, G., 2015. The adverse outcome pathway: a conceptual framework to support toxicity testing in the twenty-first century. In: Hoeng, J., Peitsch, M.C. (Eds.), Computational Systems Toxicology. Humana Press, Totowa, NJ, pp. 1–26. Rodriguez, B., Carusi, A., Abi-Gerges, N., Ariga, R., Britton, O., Bub, G., et al., 2015. Humanbased approaches to pharmacology and cardiology: an interdisciplinary and intersectorial workshop. EP Europace 18, 1287–1298. von Schomberg, R., 2007. From the Ethics of Technology Towards an Ethics of Knowledge Policy & Knowledge Assessment. A Working Document From the European Commission Services. Available at:. https://publications.europa.eu/en/publication-detail/-/ publication/aa44eb61-5be2-43d6-b528-07688fb5bd5a, Accessed date: 26 May 2017. von Schomberg, R., 2013. A vision of responsible research and innovation. Responsible innovation. In: Owen, R., Heintz, M., Bessant, J. (Eds.), Responsible Innovation: Managing the Responsible Emergence of Science and Innovation in Society. John Wiley, London, pp. 51–74. Schroeder, A.L., Ankley, G.T., Houck, K., Villeneuve, D.L., 2016. Environmental surveillance and monitoring-the next frontiers for high-throughput toxicology. Environ. Toxicol. Chem. 35, 513–525. 1556 A. Carusi et al. / Science of the Total Environment 628–629 (2018) 1542–1556Schwartz, P.J., Woosley, R.L., 2016. Predicting the unpredictable: drug-induced QT prolongation and Torsades de Pointes. J. Am. Coll. Cardiol. 67 (13):1639–1650. https://doi. org/10.1016/j.jacc.2015.12.063. Stahl, S.H., Yates, J.W., Nicholls, A.W., Kenna, J.G., Coen, M., Ortega, F., et al., 2015. Systems toxicology: modelling biomarkers of glutathione homeostasis and paracetamol metabolism. Drug Discov. Today Technol. 15, 9–14. Stilgoe, J., Owen, R., Macnaghten, P., 2013. Developing a framework for responsible innovation. Res. Policy 42, 1568–1580. Sturla, S.J., Boobis, A.R., FitzGerald, R.E., Hoeng, J., Kavlock, R.J., Schirmer, K., et al., 2014. Systems toxicology: from basic research to risk assessment. Chem. Res. Toxicol. 27, 314–329. Sullivan, K., Cochrane, S., Enoch, S., Ezendam, J., Matheson, J., Carlson, K., et al., 2017. Sensitisation of the respiratory tract induced by covalent binding of low molecular weight organic chemicals to proteins. AOP:39. https://aopwiki.org/aops/39. Taylor, K., Gordon, N., Langley, G., Higgins, W., 2008. Estimates for worldwide laboratory animal use in 2005. ATLA 36, 327–342. Tollefsen, K.E., Scholz, S., Cronin, M.T., Edwards, S.W., Knecht, J.D., Crofton, K., et al., 2014. Applying adverse outcome pathways (AOPs) to support integrated approaches to testing and assessment (IATA). Regul. Toxicol. Pharmacol. 70, 629–640. Tripathi, S., Vercruysse, S., Chawla, K., Christie, K.R., Blake, J.A., Huntley, R.P., et al., 2016. Gene regulation knowledge commons: community action takes care of DNA binding transcription factors. Database https://doi.org/10.1093/database/baw088. Villeneuve, D.L., Crump, D., GarciaReyero, N., Hecker, M., Hutchinson, T.H., LaLone, C.A., et al., 2014a. Adverse outcome pathway (AOP) development I: strategies and principles. Toxicol. Sci. 142, 312–320. Villeneuve, D.L., Crump, D., GarciaReyero, N., Hecker, M., Hutchinson, T.H., LaLone, C.A., et al., 2014b. Adverse outcome pathway development II: best practices. Toxicol. Sci. 142, 321–330.Vinken, M., Knapen, D., Vergauwen, L., Hengstler, J.G., Angrish, M., Whelan, M., 2017. Adverse outcome pathways: a concise introduction for toxicologists. Arch. Toxicol. 91, 3697–3707. Visser, S.A., de Alwis, D.P., Kerbusch, T., Stone, J.A., Allerheiligen, S.R., 2014. Implementation of quantitative and systems pharmacology in large pharma. CPT Pharmacometrics Syst Pharmacol. 3, 1–10. Wang, H., Yan, Z.G., Li, H., Yang, N.Y., Leung, K.M.Y.,Wang, Y.Z., et al., 2012. Progress of environmental management and risk assessment of industrial chemicals in China. Environ. Pollut. 165, 174–181. vanWelie, R.T.H., van Dijck, R.G.J.M., Vermeulen, N.P.E., van Sittert, N.J., 1992. Mercapturic acids, protein adducts, and DNA adducts as biomarkers of electrophilic chemicals. Crit. Rev. Toxicol. 22, 271–306. van Wenzel, A.P., Opperhuizen, A., 1995. Narcosis due to environmental pollutants in aquatic organisms: residue-based toxicity, mechanisms, and membrane burdens. Crit. Rev. Toxicol. 25, 255–279. Wittwehr, C., Aladjov, H., Ankley, G.T., Byrne, H.J., de Knecht, J., Henzie, E., et al., 2017. How adverse outcome pathways can aid the development and use of computational prediction models for regulatory toxicology. Toxicol. Sci. 155, 326–336. Woodhead, J.L., Watkins, P.B., Howell, B.A., Siler, S.Q., Shoda, L.K., 2017. The role of quantitative systems pharmacology modeling in the prediction and explanation of idiosyncratic drug-induced liver injury. Drug Metab. Pharmacokinet. 32, 40–45. Zhu, J., Wang, J., Ding, Y., Liu, B., Xiao,W., 2018. A systems-level approach for investigating organophosphorus pesticide toxicity. Ecotoxicol. Environ. Saf. 149:26–35 2018 Mar. https://doi.org/10.1016/j.ecoenv.2017.10.066. Ziman, J., 2001. Getting scientists to think about what they are doing. Sci. Eng. Ethics 7, 165–176.