applied sciences Article Value-Oriented and Ethical Technology Engineering in Industry 5.0: A Human-Centric Perspective for the Design of the Factory of the Future Francesco Longo 1,* , Antonio Padovano 1 and Steven Umbrello 2 1 Department of Mechanical, Energy and Management Engineering, University of Calabria, 87036 Rende, Italy; antonio.padovano@unical.it 2 Department of Philosophy and Educational Sciences, University of Turin, 10124 Turin, Italy; steven.umbrello@unito.it * Correspondence: francesco.longo@unical.it Received: 6 May 2020; Accepted: 16 June 2020; Published: 18 June 2020   Featured Application: This work can be used to drive the design and development of Industry 4.0 technological solutions and the Factories of the Future in a human-centric, value-oriented and ethical direction. Abstract: Although manufacturing companies are currently situated at a transition point in what has been called Industry 4.0, a new revolutionary wave-Industry 5.0-is emerging as an 'Age of Augmentation' when the human and machine reconcile and work in perfect symbiosis with one another. Recent years have indeed assisted in drawing attention to the human-centric design of Cyber-Physical Production Systems (CPPS) and to the genesis of the 'Operator 4.0', two novel concepts that raise significant ethical questions regarding the impact of technology on workers and society at large. This paper argues that a value-oriented and ethical technology engineering in Industry 5.0 is an urgent and sensitive topic as demonstrated by a survey administered to industry leaders from different companies. The Value Sensitive Design (VSD) approach is proposed as a principled framework to illustrate how technologies enabling human–machine symbiosis in the Factory of the Future can be designed to embody elicited human values and to illustrate actionable steps that engineers and designers can take in their design projects. Use cases based on real solutions and prototypes discuss how a design-for-values approach aids in the investigation and mitigation of ethical issues emerging from the implementation of technological solutions and, hence, support the migration to a symbiotic Factory of the Future. Keywords: Industry 4.0; Industry 5.0; Human-Machine Interaction; Operator 4.0; Smart Operator; ethics; technology engineering; Value Sensitive Design; smart factory; human factors 1. Introduction Recent years have witnessed technology-oriented manufacturing megatrends such as robotics, computer-integrated manufacturing and cyber-physical production systems (CPPS), molding how industrial production is seen and understood [1]. As we continue to reap the benefits of the age of 'smart production', scholars and practitioners have recently begun emphasizing the critical function that workers form as a constituent of human-centric CPPS [2], as well as the importance of designing industrial systems and work processes to integrate and embody human factors [3]. Despite numerous studies focused on the interplay between the design of work processes or workplaces and human well-being [4], technologies shaping the Factories of the Future are far more impactful. In particular, their sociotechnical influence in industrial settings has been typically overlooked, perhaps resulting Appl. Sci. 2020, 10, 4182; doi:10.3390/app10124182 www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 4182 2 of 25 from the technocratic paradigm that characterizes the 4th Industrial Revolution [5]. Although Industry 4.0 is not yet full-grown and it is still the major revolution on manufacturers' minds, some industry pioneers and visionary technology leaders have already opened up the floor to the 5th Industrial Revolution, Industry 5.0. Unlike Industry 4.0, the new revolutionary wave combines the diverging strengths of CPPS and human agents to create symbiotic factories [6]. To address this phenomenon, institutions and policy makers are shifting their attention towards 'human-centered design' as well as 'ethical and responsible innovation' in the Factories of the Future [7]. The European Union, for example, has been encouraging research into methodologies that centralize technology design for values that also encircles socio-cultural and ethical considerations [8,9]. Despite the inclination to frame technologies as neutral objects, they unavoidably manifest values, whether those of their designers or users [10], consequentially shaping and transforming how workers using them can actualize their potentials, identities, relationships, and aims [11]. Industry 5.0 is compelling computer scientists, designers, industrial engineers, as well as philosophers and legal experts to concentrate on the means by which technologies within 5.0 industrial systems can be designed for human values, rather than relegating them as an afterthought [12]. This is particularly apparent when converging technologies blur the lines between human and technological capabilities [13], such as in the case of the emerging Operator 4.0 paradigm. Scholars [13] have recently taken up the invitation to find acceptable and ethically sustainable solutions together with the employees. Given the impacts that manufacturing systems have on both workers and society, coupled with the fact that the human-centric CPPS paradigm is still in its infancy, it warrants investigating the role of human values and ethical design for the converging technologies enabling the Operator 4.0. Scope and Aims This paper argues that a 5th revolutionary wave in the industrial sector, Industry 5.0, is imminent and develops in tandem with a growing consciousness of the value of a human-machine symbiosis in industry. The emergence of Industry 5.0 as the Age of Augmentation is discussed in Section 2 where ethical questions are presented regarding the impact of technology on human workers in the field of future industrial systems engineering. The second aim of this study is to ascertain whether the industrial community is truly attentive to the human-centric evolution of the Industry 4.0 paradigm towards an Industry 5.0 and whether human values and ethical issues represent an actual concern for the workers at different levels and in different kind of organizations. A survey has been administered to industry leaders from different companies already leading the 4th industrial revolution to different extents and the results are discussed in Section 3. What makes this paper comparatively unique in its approach is that it then proposes a Value Sensitive Design (VSD) methodology as an overarching framework for a human-centered design of the Factories of the Future, with technologies shaping and embodying human values. Guidelines and recommendations are provided in Section 4 that are based on the results of the survey, including a list of values-that may serve as starting point for future studies-to uphold during techno-social change and human-machine symbiosis towards the Industry 5.0. Three use cases are finally proposed in Section 5 and discussed. The use cases build upon technological solutions and prototypes for the Operator 4.0 and describe prospective scenarios that industrial workers will deal with in the Factories of the Future. Insights describe how a design for values aids in the investigation and mitigation of ethical issues concealed behind the implementation of specific technological solutions and, hence, support the migration to a symbiotic Industry 5.0. Finally, Section 6 draws the conclusions by summarizing the contribution of the study, illustrates the limitations and suggests directions for future research. Appl. Sci. 2020, 10, 4182 3 of 25 2. Industry 5.0 2.1. The Emergence of Industry 5.0 as the Age of Augmentation Almost a decade after the term Industry 4.0 has been coined and the benefits of an interconnected, highly digitalized, and automated smart factory are well-documented in the literature, most industrial firms are still 'transitioning' to the smart manufacturing paradigm, rather than 'revolutionizing' [14]. They are currently experimenting and piloting standalone solutions to establish a digital foundation that may yield in the short or medium term greater efficiency into traditional manufacturing routines [15]. Despite the fact that the 4th industrial revolution seems to set aside the human component, the ubiquitous integration of digital and automation technologies in the industrial socio-technical framework has today spontaneously triggered the genesis of a futuristic vision of a new evolutionary stage where the human and machine reconcile and work in perfect symbiosis with one another (Figure 1). After the last two 'industrial ages'-the Age of Automation (Industry 3.0) and the Digital Age (Industry 4.0)-where smarter and smarter connected devices and computer technologies have automated manufacturing processes and workflows, the next 'age'-the Age of Augmentation (Industry 5.0)-will be focused on the cooperation between human intelligence and cognitive computing and on treating automation as a further enhancement of the human's physical, sensorial, and cognitive capabilities [16]. By putting humans back into the loop, Industry 5.0 profoundly restructures human tasks in the realm of manufacturing in ways that benefit the workers. They will be upskilled to shift from manual to cognitive labor [17], to provide value-added tasks in production and to work-with peace of mind-alongside an autonomous workforce, i.e., collaborative robots, that will be perceptive and informed about human intention and desire [18]. Appl. Sci. 2020, 10, x FOR PEER REVIEW 3 of 25 establish a digital foundation t t may yield in the short or medium term greater efficiency into 93 traditional manufacturing routines [15]. Desp te he fact that the 4th industrial revolutio seems to set 94 aside the human component, the ubiquit us i tegration of digital and automati n technologies in the 95 industrial socio-technical framework has today spontaneously triggered the genesis of a futuristic 96 vision of a new evolutionary stage where the human and machine reconcile and work in perfect 97 symbiosis with one another (Figure 1). After the last two 'industrial ages'-the Age of Automation 98 (Industry 3.0) and the Digital Age (Industry 4.0)-where smarter and smarter connected devices and 99 computer technologies have automated manufacturing processes and workflows, the next 'age'-the 100 Age of Augmentation (Industry 5.0)-will be focused on the cooperation between human intelligence 101 and cognitive computing and on treating automation as a further enhancement of the human's 102 physical, sensorial, and cognitive capabilities [16]. By putting humans back into the loop, Industry 103 5.0 profoundly restructures human tasks in the realm of manufacturing in ways that benefit the 104 workers. They will be upskilled to shift from manual to cognitive labor [17], to provide value-added 105 tas s in production and to work-with peace of ind-alongsid an autonomous workf rce, i.e. 106 collaborative robots, that will be perceptive and informe about human intention and desire [18]. 107 108 109 Figure 1. Moving to the fifth industrial revolution or Industry 5.0: the Age of Augmentation. 110 Although this discourse may appear premature and Industry 5.0 literature does not exist yet, 111 the 5th industrial revolution is already underway among the enterprises that are just now adopting 112 the principles of Industry 4.0. Recent years have indeed assisted in drawing attention to the human-113 centric design of CPPS [19] and to the genesis of the 'Operator 4.0', defined as a hybrid agent 114 established as the product of a symbiotic relationship between the human and the machine [20]. The 115 impact of the technologies enabling the Operator 4.0 on human perception, cognition and interaction 116 capabilities is explained in Figure 2. Coined as an enabler for computers to perceive the world, the 117 Internet of Things (IoT) paradigm evolved now full circle to aim at enriching the human perception 118 of the CPPS (and of the environment in general) with a multitude of sensing devices and technologies. 119 Nonetheless, the abundance of data generated by such devices has not been yet fully exploited in a 120 production environment [21]. Augmented cognitive capabilities are needed to process, understand 121 and analyze this amount of data to get valuable information from it. Technologies enhancing 122 cognitive capabilities do not only include those belonging to the Artificial Intelligence (AI) sphere-123 i.e. cognitive computing, computer vision, knowledge representation, machine learning, 124 recommender systems and planning, scheduling and optimization algorithms. They also include 125 simulation for what-if scenario analysis, big data analytics, cloud computing (that provides on-126 demand resources, especially data storage and computing power) and virtual reality (for training 127 and experiential learning purposes). In order to move towards more cognitive and intelligent spaces 128 [22], human workers are also asked to cooperate with the CPPS and complement the robotic and 129 virtual world of the automated production system through novel technologies enabling faster and 130 more intuitive workflows [23]. A considerable amount of information and knowledge is indeed 131 generated during the augmented cognition process that should be made available to the operator in 132 a meaningful and intuitive way. Augmented Reality, mobile device and smartwatch apps and 133 advanced human–machine interfaces in industry will be augmented with voice-enabled interaction, 134 thus giving birth to digital intelligent assistants. While Industry 4.0 creates the foundation for the 135 Smart Factory, Industry 5.0 is the era of a 'Social Smart Factory', where every single cooperative 136 building block of a CPPS will be able to communicate with the human component through enterprise 137 Figure 1. Moving to the fifth industrial re r Industry 5.0: the Age of Augmentation. Although t is discourse may appear t re and Industry 5.0 literature does not exist yet, the 5th industrial revolution is already underway among the enterprises t j ti g the principles of Industry 4.0. Recent years have indeed assisted in rawing attentio to the uman-centric design of CPPS [19] and to the genesis of the 'Operator 4.0', defined as a hybrid agent established as the product of a symbiotic relationship between the human and the machine [20]. The impact of the technologies enabling the Operator 4.0 on human perception, cognition and interaction capabilities is explained in Figure 2. Coined as an enabler for computers to perceive the world, the Internet of Things (IoT) paradigm evolved now full circle to aim at enriching the human perception of the CPPS (and of the environment in general) with a multitude of sensing devices and technologies. Nonetheless, the abundance of data generated by such devices has not been yet fully exploited in a production environment [21]. Augme ted cognitive capabilities are needed to process, understand and a alyze this amount of d ta to get valuab e i forma i n from it. Technologies enha cin cognitive capabilities do not only include those belonging to he Art ficial I telligence (AI) phere-i.e., cognitive computing, computer vision, knowledge represe tation, machine learning, recommender syst ms and planning, scheduling and optimization algorithms. They also include simulation for what-if scenario analysis, big data analytics, cloud computing (that provides on-demand resources, especially data storage and computing power) and virtual reality (for training and experiential learning purposes). In order to move towards more cognitive and intelligent spaces [22], human workers are also asked to cooperate with the CPPS and complement the robotic and virtual world of the automated production system Appl. Sci. 2020, 10, 4182 4 of 25 through novel technologies enabling faster and more intuitive workflows [23]. A considerable amount of information and knowledge is indeed generated during the augmented cognition process that should be made available to the operator in a meaningful and intuitive way. Augmented Reality, mobile device and smartwatch apps and advanced human–machine interfaces in industry will be augmented with voice-enabled interaction, thus giving birth to digital intelligent assistants. While Industry 4.0 creates the foundation for the Smart Factory, Industry 5.0 is the era of a 'Social Smart Factory', where every single cooperative building block of a CPPS will be able to communicate with the human component through enterprise social networks. In the 'Social Smart Factory', humans will work alongside collaborative robots [24], will use exoskeletons to perform strenuous tasks [25] and will be able to control and act upon the CPPS remotely if needed. Given the close interplay and symbiosis between human and machine, it becomes compelling to pay attention to the possible consequences of Industry 5.0 on the welfare of humans interacting with these systems [26]. Bringing back human workers to factory floors is about creating meaningful and sustainable socio-technical systems to support harnessing smart technologies in organizations [27]. Appl. Sci. 2020, 10, x FOR PEER REVIEW 4 of 25 social networks. In the 'Social Smart F ctory', humans will work alongside coll b rative robots [24], 138 will use exoskel tons to p rform str nuous tasks [25] and will be ble to control and a t up n the 139 CPPS remotely if needed. Given the close interplay and symbiosis between human and machine, it 140 becomes compelling to pay attention to the possible consequences of Industry 5.0 on the welfare of 141 humans interacting with these systems [26]. Bringing back human workers to factory floors is about 142 creating meaningful and sustainable socio-technical systems to support harnessing smart 143 technologies in organizations [27]. 144 145 146 Figure 2. Towards human–machine symbiosis in the Industry 5.0: technology-augmented human 147 capabilities as part of a cognitive process. 148 2.2. The Role of Ethics in Industry 5.0 149 Since organizations have witnessed an increasing human–machine cooperation level, more and 150 more ethical questions and concerns have been raised regarding the impact of technology on humans 151 in the field of future industrial systems engineering [28]. Just like ethics acts as a self-governing 152 system to keep human self-interest and the good of society in equilibrium, so we argue that ethics is 153 expected to fuel a symbiotic relationship between humans and the cyber-physical world in Industry 154 5.0. Although literature specifically focusing on ethical issues related to technologies is steadily 155 expanding, much of this research focuses on separate individual values or technologies. A significant 156 body of research focuses, for example, on ethical concerns related to information and communication 157 technologies (ICT) and robotic engineering. Human–robot co-working issues have been, for example, 158 identified in psychological issues, lack of social interaction, skepticism towards learning robots, 159 shrinking human workforce, and human–robot competition, among others [29]. Although this 160 research is very valuable to flesh out specific values, it generally does not take a holistic approach. 161 This can be a miss particularly when values are competing. Hence, even if the theoretical background 162 exists and face-to-face interviews are widely adopted to elicit values influencing the future of 163 industrial workforce and its connection to technology [30], current research in this field remains 164 scarce and most of the research efforts focus asymmetrically on the technical aspects of the Factories 165 of the Future. 166 In Industry 5.0, instead, researchers are compelled to design CPPS using a 'human-centered' 167 rather than a 'techno-centered' approach [26], with a focus on social and ethical considerations [5]. 168 The vision of Industry 5.0 comprises of two prospective scenarios, commonly known as the 169 automation and the tooling scenarios, which are characterized by a different degree of self-X abilities 170 of the CPPS [31]. On the one hand, the 'automation' scenario proposes an extensive automation of 171 Figure 2. Towards human–machine symbiosis in the Industry 5.0: technology-augmented human capabilities as part of a cognitive process. 2.2. The Role of Ethics in Industry 5.0 Since organizations have witnessed an increasing human–machine cooperation level, more and more ethical questions and concerns have been raised regarding the impact of technology on humans in the field of future industrial systems engineering [28]. Just like ethics acts as a self-governing system to keep human self-interest and the good of society in equilibrium, so we argue that ethics is expected to fuel a symbiotic relationship between humans and the cyber-physical world in Industry 5.0. Although literature specifically focusing on ethical issues related to technologies is steadily expanding, much of this research focus s on separate individual al es or technologies. A significant body of research focuses, for example, on ethical concerns related i formation and communication technologies (ICT) and robotic engine ring. –robot co-working issues hav been, for example, identified in psychol gical issues, lack of s cial i teraction, skepticism towards learning robots, shrinking human workforce, and human–robot competition, among others [29]. Although this research is very valuable to flesh out specific values, it generally does not take a holistic approach. This can be a miss particularly when values are competing. Hence, even if the theoretical background exists and face-to-face interviews are widely adopted to elicit values influencing the future of industrial workforce and its connection to technology [30], current research in this field remains scarce and most of the research efforts focus asymmetrically on the technical aspects of the Factories of the Future. Appl. Sci. 2020, 10, 4182 5 of 25 In Industry 5.0, instead, researchers are compelled to design CPPS using a 'human-centered' rather than a 'techno-centered' approach [26], with a focus on social and ethical considerations [5]. The vision of Industry 5.0 comprises of two prospective scenarios, commonly known as the automation and the tooling scenarios, which are characterized by a different degree of self-X abilities of the CPPS [31]. On the one hand, the 'automation' scenario proposes an extensive automation of work processes and tasks, including planning of production process that might eventually lead to a devaluation of simple working tasks and to maintenance and supervision efforts performed only by highly qualified personnel [32]. Conversely, in the 'tooling' scenario, technology is mostly implemented to extend human capabilities and support the worker to make the right decision [16]. Which of the two scenarios will be dominant is not predictable and will depend on the pace of technological advances, and company-related conditions among other factors. However, in both cases, serious ethical harms can arise because of either poor design, inappropriate application or misuse of multiple types of technologies in industrial environments. Two sub-fields of applied ethics are directly relevant: (1) how to design industrial systems based on ethical principles and (2) the ethics of the technologies themselves, or machine ethics. Due to greater intelligence embedded into production systems, machine ethics-i.e., how these systems can themselves behave ethically-is becoming relevant, but further discussion of this subject is not within the scope of this article. Instead, the role of industrial systems designers is to examine human values and ethical aspects a priori so that they are not construed as costs, but instead as design requirements. Designers have traditionally employed cost-benefit analyses in determining design requirements where, for example, safety and efficacy are at odds and safety is usually categorized as a cost [33]. Today, this approach is no longer acceptable. Any poorly value-oriented design can result in a lack of technological transformation due to barriers for the adoption of technologies [34] or serious ethical issues. Besides the huge investments required for the current industrial transformation, companies are experiencing significant worker hesitation to adopt Industry 4.0 technologies. This might be intended as a genuine concern due to the lack of a robust ethical framework that encompasses all the aspects of the Factory of the Future. 3. Ethics and Technology in Industrial Enterprises: A Survey In order to ascertain to what extent the industrial community is attentive to the human-centric evolution of the Industry 4.0 paradigm towards an Industry 5.0, and whether human values and ethical issues represent a barrier for adoption and an actual concern for the workers, a survey has been administered to industry leaders from different companies already leading the 4th industrial revolution at different levels and with different purposes. 3.1. Survey Structure and Administration A Scalable Information Dimension (SID) and Value-oriented Semi-structured Survey [35] has been constructed to tap into respondents' understandings, views and values about the technologies adopted to augment the capabilities of the workers as well as to tease apart the influence of a technology's pervasiveness, proximity, granularity of information, and other scalable dimensions [35]. After the opening section for the collection of anonymous socio-demographic data, the survey comprises of four sections with a total of 42 questions. The questions delve into their prior knowledge and opinions about a large set of technologies enabling today the human–machine symbiosis in industrial environments, examine the interest of their enterprises in these kinds of technologies, and elicit human values and ethical questions related to the augmentation of workers' capabilities in a hypothetical Industry 5.0 scenario. For the sake of simplicity, technologies included in the survey were identified with the following acronyms: Activity and Motion Trackers (AT); Artificial Intelligence, including all its subfields (AI); Augmented Reality (AR); Big Data (BD); Cloud Computing (CC); Collaborative Robots (COB); Digital Voice-enabled Assistants (DA); Exoskeletons (EXO); Health Monitoring Devices (HMD); Human-Machine Interface (HMI); Internet of Things, including machine sensors and identification Appl. Sci. 2020, 10, 4182 6 of 25 sensors, e.g., RFID (IOT); Mobile Devices, e.g., smartphone or tablets (MD); Simulation (SIM); Social Networks, including tools for collaborative working and smart interaction with work equipment (SN); Teleoperated Systems, including devices for remote supervision and control, e.g., PLCs, actuators (TOS); Virtual Reality (VR); Wearable Devices, including smart clothes and smart watches (WD); and Work Environment Sensors, including visual, temperature or other sensors (WES). In order to aid the elicitation process, this survey applies Schwartz's 10 values to the workplace, thus defining a potential map of work values that guide individuals in work-related activities [36]. Items were created in the form of brief statements which reveal diverse aspects of each value type (three items per dimension, for a total of 30 items) and describe an example concrete work situation considered important by a hypothetical person (e.g., 'for him/her it is important . . . to succeed more than others'). Participants were asked to evaluate the extent to which they view themselves as similar to the person in the item on a 5-point Likert scale (from 1 = 'Not at all like me' to 5 = 'Very much like me'). To enhance the reliability of the survey, in the other questions, the same values were proposed by using different wording and synonyms (e.g., the value 'Achievement' was replaced by 'recognition of one's abilities and results'). Furthermore, throughout the value section of the survey, the values from VSD [37] that are considered important for technological design were also proposed to the respondents as an input for the elicitation, namely Welfare, Ownership and Property, Privacy, Freedom from Bias, Universal Usability, Trust, Autonomy, Informed Consent, Accountability, Courtesy, Identity, Calmness and Environmental Sustainability. The survey was designed in Google Forms for its ease of access and use, and distributed via email to industry experts. At the beginning of the survey, the respondents were informed of the survey's purposes, terms and conditions. Thereafter, the Operator 4.0 concept and the technologies enabling human–machine symbiosis in the Factory of the Future were briefly presented to uniform their interpretation of the context under study. Illustrative images of mock-ups, narratives, and stories of use of 4.0 technologies as well as videos of workers using prototypes of such technologies in daily situations were also used to investigate value implications of technologies that are yet to be built or widely adopted. An example video that shows one of the prototypes of a voice-enabled knowledge assistant on the factory floor used for the survey is provided in the supplementary materials. 3.2. Participants About a hundred different-sized enterprises, with at least one Italian headquarter and currently transitioning (according to the press and their website information) from Industry 3.0 to Industry 4.0, were selected from several industry sectors, including Manufacturing, Industrial Automation, Electronics, Oil&Gas, Energy, Shipping, Waste Management, Food, Fashion, Construction, and ICT. ICT enterprises were included in this survey, albeit not strictly related to the industrial and manufacturing domain, given their role in designing new technologies. Eighteen enterprises responded to the survey: three microenterprises (fewer than 10 employees), eight small enterprises (10 to 49 employees), two medium-sized enterprises (50 to 249 employees), and five large enterprises (250 or more employees). Reports from different online survey platforms demonstrated that surveys with lower response rates (near 20%) had more accurate measurements compared to surveys with higher response rates (near 60 or 70%) [38]. The answers' accuracy was crosschecked in this study during some follow-up meetings and emails with the respondents. It is also important to mention that the aim of the survey is not to carry out a statistical analysis over a huge dataset, but to understand which is the first feeling and perception of industry leaders in different industrial sectors about the topics of the Operator 4.0 (or Augmented Operator) as well as Value-oriented and Ethical Technology Engineering. The survey, which begins with a section gathering anonymous socio-demographic data, was answered by 18 people (83% males), ranging in age from 28 to 64 and distributed as follows: 11.1% between 25–34 y.o., 38.9% between 35–44 y.o., and 22.2% between 45–54 y.o. The majority of the respondents (≈ 72%) hold a managerial role, while the remaining ones are operatives, specialists and technicians. Their professional experience is as follows: 5.6% less than 5 years, 22.2% between Appl. Sci. 2020, 10, 4182 7 of 25 5–10 years, 27.8% between 11–15 years, 16.7% between 16–20 years, 11.1% between 21–25 years, 5.6% between 26–30 years, and 11.1% more than 30 years. With respect to their level of education, the 61.1% reported to have a master degree, the 22.2% a bachelor's degree, while the remaining ones have other post-graduate education (e.g., Ph.D. or Professional Master). 3.3. Survey Results and Discussion 3.3.1. Section 1: Prior Knowledge The human-centric design of the Factory of the Future and the Operator 4.0 rose in popularity over the last few years in the industrial community. Only 11.1% of the respondents never heard about this concept, while the remaining respondents heard about this shift towards a human-centered industry from meetings, fairs, and conferences, but also newspapers and political talks. The 72.2% of the surveyed companies have already initiated a roadmap for the transformation of the worker into an Operator 4.0, while all the respondents reported that the management of their enterprises show an obvious interest to implement and use such technologies. It is notable that technologies enabling human–machine symbiosis in an industrial context do not receive the same level of attention. A snapshot of the digital transformation of the enterprises under study in this work is presented in Figure 3. While all the technologies are widely known by the respondents, only a few of them are extensively used in the workplace. Among others, IOT, AI, CC, and MD receive the greatest attention at the executive level, probably for their obvious return on investment. Nevertheless, executives and managers still have limited confidence with such technologies and are skeptical about their real benefits to the business. For this reason, most of them have started to implement them at the workplace as piloting standalone solutions developed in collaborations with research institutions. Furthermore, it can be noted that technologies such as AT, EXO and HMD are not used at the workplace. While EXO are characterized by notable investment costs and still few beneficial applications were documented, AT and HMD imply significant privacy concerns for the workforce, thus showing how a discussion over the ethical issues connected to the use of technology and their design for values is an imperative in Industry 5.0. Appl. Sci. 2020, 10, x FOR PEER REVIEW 7 of 25 3.3.1. Section 1: Prior Knowledge 271 The human-centric design of the Factory of the Future and the Operator 4.0 rose in popularity 272 over the last few years in the industrial community. Only 11.1% of the respondents never heard about 273 this concept, while the remaining respondents heard about this shift towards a human-centered 274 industry from meetings, fairs, and conferences, but also newspapers and political talks. The 72.2% of 275 the surveyed companies have already initiated a roadmap for the transformation of the worker into 276 an Operator 4.0, while all the respondents reported that the management of their enterprises show 277 an obvious interest to implement and use such technologies. It is notable that technologies enabling 278 human–machine symbiosis in an industrial context do not receive the same level of attention. A 279 snapshot of the digital transformation of the enterprises under study in this work is presented in 280 Figure 3. hile all the technologies are widely known by the respondents, only a few of them are 281 extensively used in the orkplace. Among others, IOT, AI, C , and MD receive the greatest attention 282 at the executive l l, r l for their obvious return on investment. Nev rtheless, executives and 283 managers still have limited confidence with such technologies and are skeptical about th ir r al 284 ben fits to the busines . For this reas n, most of them have started to implemen them at the 285 w rkplace as piloting standalone solutions developed in collaborations with research institutions. 286 Furtherm r , it can be noted that technologies such as AT, EXO and HMD are not used at the 287 workplace. While EXO are characterized by notable investment costs and still few beneficial 288 applications were documented, AT and HMD imply significant privacy concerns for the workforce, 289 thus showing how a discussion over the ethical issues connected to the use of technology and their 290 design for values is an imperative in Industry 5.0. 291 292 0% 20% 40% 60% 80% 100% AI AR AT BD CC COB DA EXO HMD HMI IOT MD SIM SN TOS VR WD WES Percentage of respondents w ho knew the technology Percentage of respondents w ho used the technology at the w orkplace Percentage of enterprises that have adop ted the technology Percentage of enterprises w here the technology is considered critical at the executive level 293 Figure 3. Prior knowledge and use of the Operator 4.0 enabling technologies. 294 3.3.2. Section 2: Opinion 295 Despite a certain skepticism, the respondents unanimously agree on the prospective beneficial 296 impact of augmenting the worker's capabilities. What emerges from the survey in the first place is 297 the importance to extend the worker's cognitive capabilities to fill the gap with technology (they had 298 a 4.1 score on a 5-point Likert scale). The capability to interact with other workers and with the CPPS 299 in an intuitive and smart manner resulted to be almost as important as the 'cognitive' one (the 300 Figure 3. Prior knowledge and use of the Operator 4.0 enabling technologies. Appl. Sci. 2020, 10, 4182 8 of 25 3.3.2. Section 2: Opinion Despite a certain skepticism, the respondents unanimously agree on the prospective beneficial impact of augmenting the worker's capabilities. What emerges from the survey in the first place is the importance to extend the worker's cognitive capabilities to fill the gap with technology (they had a 4.1 score on a 5-point Likert scale). The capability to interact with other workers and with the CPPS in an intuitive and smart manner resulted to be almost as important as the 'cognitive' one (the 'interaction capabilities' scored 3.8), unlike the 'sensorial capabilities', whose augmentation is considered mildly important by the respondents (3.5 score). A more detailed view on this segmentation is given in the treemap chart of Figure 4 that is a qualitative illustration to compare proportions within groups of technologies augmenting human's capabilities. In this chart, each technology is represented by a rectangular shape, where bigger rectangles indicate a higher perceived beneficial impact of the technology by the industry expert. The color allows to distinguish the categories of human capabilities (i.e., cognitive, interaction and sensorial) that the technology augments and to understand the significance of each technology set. This result is in line with the literature and the data from the previous section that pictures the current situation of the enterprises' digital transformation. Indeed, studies focusing on the augmentation of the worker's cognitive capabilities in industrial environments recently emerged in the perspective of the 'tooling' scenario. The survey respondents pointed out several prospective benefits of implementing such technologies, especially at the field level. Human–robot collaboration would increase production efficiency, whereas an enhanced problem solving (e.g., AR tools may help maintenance technicians to minimize machine stop time) is enabled by a faster and intuitive access to data (e.g., workers' health data), information, competences, and knowledge available at the workplace. In their opinion, maintenance, manufacturing and process supervision tasks will be impacted the most, but they do also mention improvements for the capability to respond to market changes, for planning activities, manual assembling, process control, quality checks, as well as back-office activities if such technologies are integrated with enterprise systems (e.g., MES). Appl. Sci. 2020, 10, x FOR PEER REVIEW 9 of 25 339 Figure 4. Treemap chart of the perceived beneficial impact of Operator 4.0 enabling technologies. 340 3.3.3. Section 3–4: Human Values and Ethics 341 The same inclination holds for when the respondents agreed to the fact that the technologies that 342 the workers use (or will use) in the Factory of the Future should embed or be designed for human 343 values: 88.9% responded 'yes, definitely' while the remaining responded 'maybe'. The majority of the 344 surveyed enterprises revealed to be very attentive to human values and formal ethical codes of 345 conduct are available in 72% of the cases, especially in the larger enterprises. 346 However, when it comes to technology, respondents are even more attentive to human values, 347 as demonstrated in Figure 5. The questionnaire items analysis revealed that, while the perceived level 348 of importance for some values does not change when associated to technology, it significantly rises 349 for some others if compared to their personal value scale, such as in the case of 'conformity', 350 'tradition', 'achievement', but also 'universalism' and 'security'. This result captures the conflict 351 between the disruptive effect of technology and the workers' inclination to preserve the past and 352 resist change in the workplace in order to maintain the status quo. The 'security' value confirms this 353 interpretation, as workers-especially older adults-strive to get stability and safety in their 354 workplace (something that the introduction of new technologies cannot guarantee in their opinion). 355 In contrast to the 'conservation' mindset, workers-in particular younger adults-also aim at 356 reaching goals and personal success, demonstrating their competence with technological solutions. 357 The analysis of the extent to which values from VSD should be considered when designing new 358 technologies and solutions for the factory workers confirmed the findings above. All the values were 359 rated on a range between 3.45 (for 'Ownership & Property') and 4.36 (for 'Trust'). It proves that 360 respondents are concerned about the fact that technology should be trustful, serve for the common 361 good and be designed to prevent any possible misuse. Linked to the need of digital trust, the 362 respondents cited 'honesty', 'explainability' and 'transparency' as well as 'integrity' and 363 'professionality' as the foremost values that any technology used by the Operator 4.0 at the workplace 364 should embed. In fact, the analysis of a list of seven ethical issues proposed to the respondents-365 including unethical conduct (lying, deception, theft,...), toxic workplace culture, discrimination and 366 harassment, privacy and confidentiality, unethical leadership, unrealistic and conflicting goals, and 367 misuse of technology-corroborated the value of the ethical conduct. Despite the fact that all the 368 issues scored a mean value between 3.36 and 4.09 on a 5-point Likert scale (thus demonstrating the 369 importance of a design for ethics), digital and automation technology is explicitly considered as a tool 370 that should support accountability and responsibility. This may be seen somehow in contrast to the 371 values of second-most importance for the respondents, namely informed consent, privacy and 372 confidentiality. 373 Figure 4. Treemap chart of the perceived beneficial impact of Operator 4.0 enabling technologies. Another significant aspect emerging from this section of the survey refers to the beneficiaries of the current 4.0 digital transformation and explicitly calls for an Industry 5.0. Based on the feedback of the industry leaders, the technologies under study are expected to positively impact (e.g., by simplifying) the work of technicians, deskless operators, warehouse workers, and field specialists, but also production and maintenance managers/supervisors. However, they are also the ones that are bearing (or will bear) the negative im acts of this transformation. In the end, a paradox is established: the workers, whose work may get the highest benefits, are also the ones that may observe a deterioration of their Appl. Sci. 2020, 10, 4182 9 of 25 work conditions, or even lose their job if the 'tooling' scenario evolves in an uncontrolled manner towards the 'automation' scenario. Among others, the respondents reported significant implications, such as the worker's alienation, a higher control on working times and duties, with consequent reduced freedom and autonomy for the workers. The collection of personal data increases privacy issues, thus causing resistance to change especially by workers with an old mindset or with work-related stress. Despite the fact that the workforce downsizing risk is of course the most predominant potential negative aspect of Industry 4.0, the majority of the workers in the responding enterprises (≈89%) are resolutely willing to undertake this transformation. It then bears investigating the role of human values and ethical design for the converging technologies enabling a human–machine symbiosis from the perspective of a 5th industrial revolution that places the human worker again at the center of the Factory of the Future. 3.3.3. Section 3–4: Human Values and Ethics The same inclination holds for when the respondents agreed to the fact that the technologies that the workers use (or will use) in the Factory of the Future should embed or be designed for human values: 88.9% responded 'yes, definitely' while the remaining responded 'maybe'. The majority of the surveyed enterprises revealed to be very attentive to human values and formal ethical codes of conduct are available in 72% of the cases, especially in the larger enterprises. However, when it comes to technology, respondents are even more attentive to human values, as demonstrated in Figure 5. The questionnaire items analysis revealed that, while the perceived level of importance for some values does not change when associated to technology, it significantly rises for some others if compared to their personal value scale, such as in the case of 'conformity', 'tradition', 'achievement', but also 'universalism' and 'security'. This result captures the conflict between the disruptive effect of technology and the workers' inclination to preserve the past and resist change in the workplace in order to maintain the status quo. The 'security' value confirms this interpretation, as workers-especially older adults-strive to get stability and safety in their workplace (something that the introduction of new technologies cannot guarantee in their opinion). In contrast to the 'conservation' mindset, workers-in particular younger adults-also aim at reaching goals and personal success, demonstrating their competence with technological solutions. Appl. Sci. 2020, 10, x FOR PEER REVIEW 10 of 25 Autonomy, courtesy, calmness, and universal usability also scored more than four among the 374 VSD values. As emerged from the analysis of the Schwartz's values, 'universality' should also be a 375 core value in the human-centered design of the Factories of the Future. The human–machine 376 symbiosis in Industry 5.0 should be fair and respectful of everyone's dignity and opinion equally. It 377 should not create relational gaps, but foster social interaction: the respondents frequently revealed 378 their concern about worker alienation and work depersonalization that an ethical design of the 379 human–machine symbiosis in Industry 5.0 should avoid. In this case, poor technology design that 380 does not res ect the roles and expertise of a person in the company may egatively impact the value 381 of xperi nced people a , ultimately, lead to job displacement. Indeed, some respondents argue that 382 technology may ov rpowe the workers, for example, if technology is des gned w th episodic 383 memory like hum ns, with he aim to improve the workers' problem solving skills. However, 384 techn logy may also give some workers (e.g. younger ones) the opportunity to overpower 385 (consciously or not) other people (e.g. older ones). Instead, value-oriented technology should be 386 centered on the workforce's continuous learning and growth thanks to a sharing and collaboration 387 mindset (for example, knowledge and expertise). It is eventually opportunity for the executives 388 and managers to express a 'positive leadership' and to establish a healthy work envir nment and 389 workplace culture. 390 Similarly to Figure 4, the treemap chart of Figure 6 is a qualitative illustration to make a 391 comparison between and within groups of technologies augmenting human's capabilities where the 392 size of a rectangular shape is proportional to the respondents' concern and opinion that the 393 technology implies bigger ethical issues and challenges. It shows that the respondents expressed 394 ethical concerns mainly towards BD and AI, while SN, DA, HMI, and AT were of second-most 395 criticality. Although the ethical issues connected to these technologies are quite well-known, the low 396 awareness towards the potential ethical issues that may derive from the use of co-bots at workplace 397 is a surprising result. 398 It can be concluded that, despite some skepticism by small enterprises, all the respondents 399 reported that the workers are aware of the ethical issues in the workplace and agree on the fact that 400 a value-oriented design of a future human–machine symbiosis in industrial organizations may aid in 401 the amelioration of the ethical issues associated with technologies and lead the enterprises towards a 402 new industrial age, Industry 5.0. 403 404 Figure 5. Perceived importance of the Schwartz's ten values. 405 Figure 5. Perceived i portance of the Schwartz's ten values. Appl. Sci. 2020, 10, 4182 10 of 25 The analysis of the extent to which values from VSD should be considered when designing new technologies and solutions for the factory workers confirmed the findings above. All the values were rated on a range between 3.45 (for 'Ownership & Property') and 4.36 (for 'Trust'). It proves that respondents are concerned about the fact that technology should be trustful, serve for the common good and be designed to prevent any possible misuse. Linked to the need of digital trust, the respondents cited 'honesty', 'explainability' and 'transparency' as well as 'integrity' and 'professionality' as the foremost values that any technology used by the Operator 4.0 at the workplace should embed. In fact, the analysis of a list of seven ethical issues proposed to the respondents-including unethical conduct (lying, deception, theft, . . . ), toxic workplace culture, discrimination and harassment, privacy and confidentiality, unethical leadership, unrealistic and conflicting goals, and misuse of technology-corroborated the value of the ethical conduct. Despite the fact that all the issues scored a mean value between 3.36 and 4.09 on a 5-point Likert scale (thus demonstrating the importance of a design for ethics), digital and automation technology is explicitly considered as a tool that should support accountability and responsibility. This may be seen somehow in contrast to the values of second-most importance for the respondents, namely informed consent, privacy and confidentiality. Autonomy, courtesy, calmness, and universal usability also scored more than four among the VSD values. As emerged from the analysis of the Schwartz's values, 'universality' should also be a core value in the human-centered design of the Factories of the Future. The human–machine symbiosis in Industry 5.0 should be fair and respectful of everyone's dignity and opinion equally. It should not create relational gaps, but foster social interaction: the respondents frequently revealed their concern about worker alienation and work depersonalization that an ethical design of the human–machine symbiosis in Industry 5.0 should avoid. In this case, poor technology design that does not respect the roles and expertise of a person in the company may negatively impact the value of experienced people and, ultimately, lead to job displacement. Indeed, some respondents argue that technology may overpower the workers, for example, if technology is designed with episodic memory like humans, with the aim to improve the workers' problem solving skills. However, technology may also give some workers (e.g., younger ones) the opportunity to overpower (consciously or not) other people (e.g., older ones). Instead, value-oriented technology should be centered on the workforce's continuous learning and growth thanks to a sharing and collaboration mindset (for example, knowledge and expertise). It is eventually the opportunity for the executives and managers to express a 'positive leadership' and to establish a healthy work environment and workplace culture. Similarly to Figure 4, the treemap chart of Figure 6 is a qualitative illustration to make a comparison between and within groups of technologies augmenting human's capabilities where the size of a rectangular shape is proportional to the respondents' concern and opinion that the technology implies bigger ethical issues and challenges. It shows that the respondents expressed ethical concerns mainly towards BD and AI, while SN, DA, HMI, and AT were of second-most criticality. Although the ethical issues connected to these technologies are quite well-known, the low awareness towards the potential ethical issues that may derive from the use of co-bots at workplace is a surprising result. It can be concluded that, despite some skepticism by small enterprises, all the respondents reported that the workers are aware of the ethical issues in the workplace and agree on the fact that a value-oriented design of a future human–machine symbiosis in industrial organizations may aid in the amelioration of the ethical issues associated with technologies and lead the enterprises towards a new industrial age, Industry 5.0. Appl. Sci. 2020, 10, 4182 11 of 25Appl. Sci. 2020, 10, x FOR PEER REVIEW 11 of 25 406 Figure 6. Treemap chart of the perception of the most ethically problematic technologies in industry. 407 4. Methods 408 4.1. Value Sensitive Design in the Industry 5.0 Era 409 Currently used by the European Union's Framework Programmes, Responsible Research and 410 Innovation (RRI) is founded on an interactionist principle that technology and humans co-construct 411 and co-vary one another [7]. Several design approaches to technology engineering have been 412 proposed that account for the social embeddedness of technologies and their impacts. Particular 413 emphasis is given to the involvement and elicitation of stakeholders, either those directly or indirectly 414 implicated by technology design. Approaches such as universal design [39], inclusive design [40], 415 sustainable design [41], participatory design [42], and values sensitive design [43–45], among others 416 have been theorized; Value Sensitive Design (VSD) becoming one of the most popular approaches 417 over the last 20 years [46]. Emerging from the field of human–computer interaction, VSD is precisely 418 predicated on the same premise that technology is not value-neutral, rather it is sensitive to 419 stakeholder values, whether they are direct stakeholders such as users and designers or indirect such 420 as industry CEO's, governments, and the biosphere at large [35,47]. 421 VSD is chosen as the theoretical and methodological framework to guide the design of the 422 Factory of the Future in the perspective of Industry 5.0 for two main reasons: (1) because of its 423 inherent self-reflexivity and emphasis on enrolling both direct and indirect stakeholders into the 424 design process and the philosophical investigation of implicated values, a coupling that is not 425 explicitly called for as a fundamental process in other design approaches [48]; and (2) for its 426 robustness, meaning the scope and applicability of VSD across different design spaces. VSD aims to 427 account for multiple dynamic variables ranging from a wide array of values, technologies, social 428 contexts, and stakeholder groups among others. In doing so, VSD promotes working at the meta-429 level rather than focusing on an exhaustive set of values such as safety or usability, or a particular set 430 of stakeholders such as the differently-abled. This meta-level approach divorces VSD from any 431 specific moral or theoretical vows and permits it to be widely applicable across sociodemographic 432 contexts and design spaces. To this end, the VSD methodology aims to be fundamentally proactive 433 and anticipatory, engaging with stakeholders to guide technological development in Industry 5.0 434 from an early stage and recursively self-improve throughout the design process as new values or 435 tensions emerge. This reflexivity towards beneficial development is on account of the approach's 436 tripartite structure illustrated in Figure 7. Three types of investigations-namely conceptual, 437 empirical and technical investigations-are necessarily iterative and in a continual feedback cycle 438 [49]. 439 Figure 6. Treemap chart of the perception of the most ethically problematic technologies in industry. 4. Methods 4.1. Value Sensitive Design in the Industry 5.0 Era Currently used by the European Union's Framework Programmes, Responsible Research and Innovation (RRI) i founded on an interacti i t principle that technology and humans co-construct and co-vary one another [7]. Sever l design approaches to t chnology engineeri g ave been pr posed that account for the social embeddedness of technologies and their impacts. Particular emphasis is given to the involvement and elicitat on of stakeholders, either those directly or indirectly implicated by tec nology d sign. Appr aches such as universal design [39], inclusive de ign [40], sustainable design [41], participatory design [42], and value sensitive design [43–45], among others have been theorized; Value Sensitive Design (VSD) becoming o e of th most popular approaches over the last 20 years [46]. Emerging from the field of human–computer i teracti n, VSD is precisely predic ted on the same premise that technology is not value-neutral, r ther it is sens tive to stakeholder values, whether hey are direct stakeholders such as users and designers or indi ect suc as ndustry CEO's, governments, and the biosphere at large [35,47]. VSD is chosen as the theoretical and methodological framework to guide the design of the Factory of the Future in the persp c ive of Industry 5.0 f r two main reasons: (1) because of its inherent self-reflexivity and emphasis on enrolling both direct and indirect stakeholders into the design process and th philosophical investigation of implicated values, a coupling that s not explicitly called for as a fu damental process in other design appro ches [48]; and (2) for its robustness, meaning the sco e and applicability of VSD cross different design spaces. VSD aims to account for multiple dynamic variables ranging fr m wide array of values, techn logies, social contexts, and stakeholder groups am ng others. In doing so, VSD promotes working at the meta-level rather than f cusing on an exhaustive se of valu s such a safety or usability, or a particular set of s akeh lders suc as the differently-abled. This meta-level approach divorces VSD from ny specific moral or theoretical vows and permits it to be widely applicable across sociodemographic contexts and des gn spaces. To this end, the VSD method logy aims to be fundamentally proactive and antici atory, engaging with stakeholders to guide technological dev lopment in Industry 5.0 from an earl stage and recursively se f-improve throughout the design process as new values or ensions emerge. This reflexivity towards beneficial developm nt is on account of the approach's tripartite str cture illustrated in Figure 7. Three types of inve tigations-namely conceptual, empirical and technical investigations-are ecessarily ite ative and in a continual f edback cycle [49]. Appl. Sci. 2020, 10, 4182 12 of 25 Appl. Sci. 2020, 10, x FOR PEER REVIEW 12 of 25 440 Figure 7. The tripartite iterative Value Sensitive Design methodology. 441 Not only this, but VSD encourages designers to consider always building malleable and flexible 442 architectures into technologies to permit post-deployment augmentations to be possible as new 443 values and issues emerge [12]. Previous studies have looked at how VSD can be applied to discrete 444 technologies while generally referring to their interactions with other technologies in only a cursory 445 way. To illustrate, VSD has been applied to energy systems [50,51], mobile phone usage [52], 446 architecture projects [53], and augmented reality systems just to name a few [54]. However, it works 447 speculatively too for future technologies, both near and long term. Examples include its exploratory 448 application to nanopharmaceuticals [55], molecular manufacturing [45], intelligent agent systems 449 [56], and, less futuristic, autonomous vehicles [57]. Although these studies provide useful 450 illustrations of the intricacies of how the VSD approach can be accommodated to various technologies 451 in particular, they do not provide discussions of the convergence factor of these technologies in 452 general or how they can be designed with this structural embeddedness and interdependence. 453 Furthermore, no applications of VSD to the industrial domain have been found so far in the literature. 454 Guidelines and recommendations for the three different investigations are proposed in the following 455 subsections, while use cases are illustrated in the next section. 456 4.2. Conceptual Investigations 457 The VSD approach aims to embed stakeholder values both early on and throughout the design 458 process to steer the design of the technology in such a way as to successfully map the values deemed 459 critical [12,45]. Similar to other human experiences, work experiences are 'objects' rich in meaning 460 and defined by both individual and collective values. Indeed, values contribute to ideally delineating 461 the job with respect to potential and/or favorable goals/aims to pursue. Hence, a more holistic 462 approach, taking into account sets of values (rather than individual values) is highly preferable, as it 463 allows a more comprehensive view and further balancing of values. The starting point of a conceptual 464 investigation is to look for any valuable literature containing list of values, from different ethical 465 perspectives, regardless of the context for which they are intended or in which they are used. Using 466 well-consolidated value system measures is a good starting point for this analysis. Research has 467 proposed and utilized the WVI-Work Values Inventory [58], the WIS-Work Importance Study [59] 468 and the PVQ-Portrait Values Questionnaire [60]. Beyond theoretical discussions, value lists that are 469 prominent in particular disciplines or professions (i.e. professional codes of ethics or ethical codes of 470 conduct) as well as application-based approaches (e.g. Google's AI principles) can be assessed. Lists 471 relevant for novel and emerging technologies include the anticipatory emerging technology ethics 472 list [61] or the technomoral value list [62]. Technomoral values include the following 10 values as 473 most worthy of upholding: honesty, self-control, humility, justice, courage, empathy, care, civility, 474 flexibility, perspective, magnanimity, and technomoral wisdom. Instead, the values listed in 475 anticipatory emerging technology ethics are the following: harms and risks, rights and freedoms, 476 autonomy, human dignity, privacy, property, animal rights and animal welfare, (distributive) justice, 477 Figure 7. The tripartite iterative Value Sensitive Design methodology. Not only this, but VSD encourages designers to consider always building malleable and flexible architectures into technologies to permit post-deployment augmentations to be possible as new values and issues emerge [12]. Previous studies have looked at how VSD can be applied to discrete technologies while generally referring to their interactions with other technologies in only a cursory way. To illustrate, VSD has been applied to energy systems [50,51], mobile phone usage [52], architecture projects [53], and augmented reality systems just to name a few [54]. However, it works speculatively too for future technologies, both near and long term. Examples include its exploratory application to nanopharmaceuticals [55], molecular manufacturing [45], intelligent agent systems [56], and, less futuristic, autonomous vehicles [57]. Although these studies provide useful illustrations of the intricacies of how the VSD approach can be accommodated to various technologies in particular, they do not provide discussions of the convergence factor of these technologies in general or how they can be designed with this structural embeddedness and interdependence. Furthermore, no applications of VSD to the industrial domain have been found so far in the literature. Guidelines and recommendations for the three different investigations are proposed in the following subsections, while use cases are illustrated in the next section. 4.2. Conceptual Investigations The VSD approach aims to embed stakeholder values both early on and throughout the design process to steer the design of the technology in such a way as to successfully map the values deemed critical [12,45]. Similar to other human experiences, work experiences are 'objects' rich in meaning and defined by both individual and collective values. Indeed, values contribute to ideally delineating the job with respect to potential and/or favorable goals/aims to pursue. Hence, a more holistic approach, taking into account sets of values (rather than individual values) is highly preferable, as it allows a more comprehensive view and further balancing of values. The starting point of a conceptual investigation is to look for any valuable literature containing list of values, from different ethical perspectives, regardless of the context for which they are intended or in which they are used. Using well-consolidated value system measures is a good starting point for this analysis. Research has proposed and utilized the WVI-Work Values Inventory [58], the WIS-Work Importance Study [59] and the PVQ-Portrait Values Questionnaire [60]. Beyond theoretical discussions, value lists that are prominent in particular disciplines or professions (i.e., professional codes of ethics or ethical codes of conduct) as well as application-based approaches (e.g., Google's AI principles) can be assessed. Lists relevant for novel and emerging technologies include the anticipatory emerging technology ethics list [61] or the technomoral value list [62]. Technomoral values include the following 10 values as most worthy of upholding: honesty, self-control, humility, justice, courage, empathy, care, civility, flexibility, perspective, magnanimity, and technomoral wisdom. Instead, the values listed in anticipatory emerging Appl. Sci. 2020, 10, 4182 13 of 25 technology ethics are the following: harms and risks, rights and freedoms, autonomy, human dignity, privacy, property, animal rights and animal welfare, (distributive) justice, and well-being and the common good. The recent interest towards the implementation of values into emerging technologies, such as Big Data, find application in [63], that distilled a comprehensive set of 10 ethical values for new technologies: human welfare, autonomy, non-maleficence, justice, accountability, trustworthiness, privacy, dignity, solidarity, and environmental welfare. VSD studies also have provided various lists of human values that may be important to technologies design, they typically include values such as: Welfare, Ownership and Property, Privacy, Freedom from Bias, Universal Usability, Trust, Autonomy, Informed Consent, Accountability, Courtesy, Identity, Calmness and Environmental Sustainability among others. Besides the document review of relevant literature and research projects dealing with the value-oriented design of the Factory of the Future and related technologies or the analysis of industrial practices, there are a multitude of strategies from the social sciences that establish ways of eliciting such values such as the use of stakeholder tokens, value source analysis, and value scenarios among others [64–66]. After having selected the most appropriate lists and narrowed down the spectrum of values, integration and contextualization for the specific technological solution being adopted at the workplace is the next step. It is fundamental to distil an a priori comprehensive list of values from previous conceptual work in an abstract manner. A list of values to uphold during techno-social change toward human–machine symbiosis in Industry 5.0 and recommendations for value-oriented design of technologies are presented in Table 1. The list is of course not exhaustive given that there exists no framework to provide such a list; certain values are technology-specific and some emerge over the design process and, in many cases, after the deployment of the technology [67]. Speaking about robotics, for example, [30] identified the following values: social interaction, movement and exercise, (human) autonomy, problem solving, task variety, and building with their hands. However, values may have common points of contact and are not mutually exclusive. They overlap to a certain extent; therefore, the list can be considered a priori comprehensive. Table 1. Values to uphold during techno-social change towards human–machine symbiosis in Industry 5.0. Human Value Technology Shall Be Designed to . . . Self-actualization . . . enable the workers to accomplish their own goals and obtain personal success, to demonstrate their own capabilities and competence in order to advance their career. Accountability . . . ensure full transparency of how the symbiotic CPPS operates and makes decision. It includes that the actions of a worker or group of workers is shared transparently to ensure that any problem can be traced back to the cause. Trustworthiness . . . be honest, trustworthy, reliable, and avoid systematic bias, thus performing its required function under given conditions for a stated time interval, as well as to improve interpersonal trust in a human-machine hybrid system. Privacy . . . ensure informational privacy, respect for the workers' private sphere and the right to determine what information can be communicated to others through informed consent. Welfare . . . ensure workers' health (physical well-being and peaceful psychological state) also thanks to a proper work-life balance, a balanced workload and a comfortable and pleasant work environment. Autonomy . . . allow workers to be independent, free and flexible while at work in order to express themselves, engage in autonomous thinking, make independent choices, utilize their creativity and fully use their own intelligence. Altruism . . . enhance teamwork and mutual care among the workers and to favour the wellbeing of all those one comes into contact with during one's professional activities, thus demonstrating oneself as sincere, open to helping and responsible. Appl. Sci. 2020, 10, 4182 14 of 25 Table 1. Cont. Human Value Technology Shall Be Designed to . . . Common Good . . . ensure environmental welfare and sustainability as well as to contribute to the beauty of the world and make good things. Security . . . provide a secure return (economic or abstract) to the workers and preserve order, stability and harmony within the physical environment, work-related relationships and professional activities. Stimulation . . . keep stimulating workers with challenging, various and novel tasks, that require continuous learning and professional growth and may sometimes require even a certain level of risk. Sociability . . . stimulate good social relationships and interactions with fellow workers, thus working in a social environment rather than in isolation. Identity . . . to ensure people's equality, universality, respect and inclusion and avoid discrimination of any kind. It also requires proper actions to guarantee to everyone the same possibilities of others as well as a certain amount of human control over the events. Authority . . . allow workers to attain and/or exercise a prestigious or authoritative position and/or social status that increases one's influence or ability to control other members of the organization and resources. Conformity . . . support the workers to respect rules and expectations, thus demonstrating social discipline and loyalty. At the same time, it shall restrict one's actions and/or conditioning one's choice, inclinations, impulses and desires. 4.3. Technical Investigations Technical investigations follow up on the previous analyses to determine how the technical aspects of the technology itself can support or restrict the values distilled in the previous investigation. Value Sensitive Design adopts the position that technologies in general provide value that follow from their properties. That is, a given technology is more suitable for certain activities and more readily supports certain values while rendering other activities and values more difficult to realize. In one form, the ultimate aim is to move from the values to the design requirements through the definition of norms. As mentioned, values can often be abstract philosophical concepts that are difficult to conceptualize in terms of concrete engineering requirements. Because of this, VSD aims at the embodiment of values that extend beyond-but not excluding-economic value, which is what technological solutions are traditionally designed for. Norms, which are situated in the transition point between values and design requirements, are the contextual designations of those values and can be understood as the design objectives of any given project (i.e., 'maximize safety/usability/efficiency' or 'minimize cost'). The norms of any given value are established on the basis of the industrial context-of-use in which the technology is or will eventually be deployed. To this end, the norms of any particular context can be used to derive specific design requirements that the technology can embody to support any given set of norms. Doing this appears prima facie abstract and meta-level, and for this reason we provide some salient use cases to illustrate how designers can begin to conceptualize the bi-directional process. Van de Poel is clear in saying that the 'elements at the lower levels cannot be logically deduced from the higher level elements' [68]. The reason for this is because there is no one avenue from values to design requirements; there are numerous ways that values can be translated and embodied in design requirements. In many design spaces, the successful construction of a value hierarchy will almost certainly need to be conducted in both directions. Figure 8 illustrates simply the value-norm-requirement hierarchy. Each of the norms and design requirements are not inextricably exclusive to any single value. Any of the below design requirements can just as easily satisfy other norms which in turn can satisfy other values. What engineering teams need to do is specify as best as possible how different design requirements, norms and values relate to one another in any given design project and determine how to most aptly satisfy them. Appl. Sci. 2020, 10, 4182 15 of 25 Appl. Sci. 2020, 10, x FOR PEER REVIEW 15 of 25 533 Figure 8. Bi-directional value-norm-requirement hierarchy. 534 4.4. Empirical Investigations 535 Empirical investigations conclude the iteration of the VSD cycle by enrolling stakeholders to 536 determine their relationship with the technologies in the Factory of the Future and how well the 537 identified values map on to the current design architecture [35]. As such, the functional directive of 538 this step is to enroll stakeholders directly into the design process by eliciting their values in order to 539 design technologies that more accurately mirror their values. Methods include but are not limited to 540 scalable information dimensions, value-oriented semi-structured surveys, analysis of prospective 541 working scenarios or value-oriented mock-ups and prototypes. In any case, empirical investigations 542 will eventually identify ethical issues with a bottom-up approach and answer the following 543 questions: 544  Which values should be driving technology design and development to achieve a human–545 machine symbiosis in the Factory of the Future? 546  Which values are involved with the technology at today's point in its development in industries? 547  How do those critical values in industrial workplaces align with societal priorities at large? 548  How do stakeholders prioritize competing values in design trade-offs? 549  How do they prioritize individual values and usability considerations? 550  Are there differences between espoused practice (what people say) compared with actual 551 practice (what people do)? 552  Do organizations do appropriate value considerations during the technology design and 553 implementation process? 554 This process should be reiterated every time there are technological advancements, when people 555 at the workplace change or the social context changes. This is done continually through the design 556 process. As new empirical values emerge and are defined, they may come into conflict with the a 557 priori values initially delineated in conceptual investigations, triggering a revaluation of how those 558 initial values and their technical understanding impact design. Similarly, post-deployment of any 559 given system may trigger the emergence of a previously unconsidered value, triggering a revaluation 560 of how to account for such emergence and the technical capacity for actualizing such a change. 561 Figure 8. Bi-directional value-norm-requirement hierarchy. 4.4. Empirical Investigations Empirical investigations conclude the iteration of the VSD cycle by enrolling stakeholders to deter ine their relationship with the technologies in t e Factor of the Future and how well the identified values map on to the current design architecture [35]. As such, the functional directive of this step is to enroll stakeholders directly into the design process by eliciting their values in order to design technologies that more accurately mirror their values. Methods include but are ot limited to scalable information dimensions, value-oriented semi-structured surveys, analysis of prospective working scenarios or value-oriented mock-ups and prototypes. In any case, empirical investigations will eventually identify ethical issues with a bottom-up approach and answer the following questions: • Which values should be driving technology design and development to achieve a human–machine symbiosis in the Factory of the Future? • Which values are involved with the technology at today's point in its development in industries? • How do those critical values in industrial workplaces align with societal priorities at large? • How do stakeholders prioritize competing values in design trade-offs? • How do they prioritize individual values and usability considerations? • Are there differences between espoused practice (what people say) compared with actual practice (what people do)? • Do organizations do appropriate value considerations during the technology design and implement process? This process should be reiterated every time there are technological advancements, when people at the workplace change or the social context changes. This is done continually through the design process. As new empirical values emerge and are defined, they may come into conflict with the a priori values initially delineated in conceptual investigations, triggering a revaluation of how those initial values and their technical understanding impact design. Similarly, post-deployment of any given system may trigger the emergence of a previously unconsidered value, triggering a revaluation of how to account for such emergence and the technical capacity for actualizing such a change. Appl. Sci. 2020, 10, 4182 16 of 25 5. Use Cases: Preliminary Results and Discussion Three uses cases were defined with the purpose to explore the ethical issues concealed behind the implementation of specific technological solutions towards a human–machine symbiosis in Industry 5.0. The use cases build upon solutions and prototypes developed by CAL-TEK S.r.l., a Spin-off company of the University of Calabria, Italy and carried out at the Modeling & Simulation Center-Laboratory of Enterprise Solutions (MSC-LES) at the Department of Mechanical, Energy and Management Engineering of the University of Calabria. Regarding the empirical investigations for these use cases, experts were consulted within the professional networks of the researchers involved in the projects, but also at expert forums. In particular, during a major Industry 4.0 conference, experts were asked to consider the prototypes illustrated here in the following and identify the values from Table 1 that are more relevant or identify new ones. In most instances, experts confirmed that the proposed list of values was comprehensive. The discussions with them were particularly valuable in deepening the understanding of and thinking on how to regard particular values when designing technological solutions for achieving human–machine symbiosis in the context of Industry 5.0. 5.1. Use Case #1: Augmenting the Workers' Perception Capabilities ->Monitoring Workers' Health and Movements via Wearable Sensors Wearable technology, including smart garments (e.g., gloves) wristbands and smartwatches, are designed to collect the data of workers' personal health as well as to track the movement of the operators. Prototypes were developed to track the movements of the worker's arms and fingers and, at the same time, to collect data about their heart rate variability to monitor the task-related stress during manual assembling tasks in a manufacturing company (Figure 9). It is worth mentioning that using these devices requires handling and processing big data, therefore, the developed solution embeds big data analytics methodologies. Appl. Sci. 2020, 10, x FOR PEER REVIEW 16 of 25 5. Use Cases: Preliminary Results and Discussion 562 Three uses cases were defined with the purpose to explore the ethical issues concealed behind 563 the implementation of specific technological solutions towards a human–machine symbiosis in 564 Industry 5.0. The use cases build upon solutions and prototypes developed by CAL-TEK S.r.l., a Spin-565 off company of the University of Calabria, Italy and carried out at the Modeling & Simulation Center-566 Laboratory of Enterprise Solutions (MSC-LES) at the Department of Mechanical, Energy and 567 Management Engineering of the University of Calabria. 568 Regarding the empirical investigations for these use cases, experts were consulted within the 569 professional networks of the researchers involved in the projects, but also at expert forums. In 570 particular, during a major Industry 4.0 conference, experts were asked to consider the prototypes 571 illustrated here in the following and identify the values from Table 1 that are more relevant or identify 572 new ones. In most instances, experts confirmed that the proposed list of values was comprehensive. 573 The discussions with them were particularly valuable in deepening the understanding of and 574 thinking on how to regard particular values when designing technological solutions for achieving 575 human–machine symbiosis in the context of Industry 5.0. 576 5.1. se ase #1: g e ti g the orkers' erceptio apabilities -> o itori g orkers' ealth a d 577 i l s rs 578 l i l i g smart garments (e.g. gloves) wristbands and smartwa ches, are 579 designed to c llect the data of workers' personal health as well as to track the t 580 . fi 581 ll t i l 582 . 583 , t , t l l t 584 l i . 585 586 587 Figure 9. Example of wearable technology-from left to right, wristbands, gloves, smartwatches-588 augmenting workers' perception capabilities. 589 Figure 10 lays out one potential way to translate values, through norms and into design 590 requirements that embody the value of welfare. The value was considered important in the 591 environments in which the augmented operator works. As mentioned above, the bi-directional 592 hierarchy of values does not prescribe one direction over the other, but instead allows for various 593 permutations to be undertaken depending on the context of design/use and the current design 594 paradigms that are specific to the context. Because of this, various design requirements can ultimately 595 satisfy the final value(s). 596 597 i re 9. le f e r le tec l fr left t ri t, rist s, l es, s rt tc es a e ti or ers' erce tio ca a ilities. Figure 10 lays out one potential way to translate values, through norms and into design requirements that embody the value of welfare. The value was considered important in the environments in which the augmented operator works. As mentioned above, the bi-directional hierarchy of values does not prescribe one direction over the other, but instead allows for various permutations to be undertaken depending on the context of design/use and the current design paradigms that are specific to the context. Because of this, various design requirements can ultimately satisfy the final value(s). Appl. Sci. 2020, 10, 4182 17 of 25 Appl. Sci. 2020, 10, x FOR PEER REVIEW 17 of 25 598 Figure 10. Bi-direction hierarchy showcasing how the value of welfare can be translated through 599 norms and into technical design requirements for wearable technology. 600 While values such as welfare have already some background literature (e.g. ergonomics of 601 wearable devices) that may help the designers, some other values are more intricate, for example the 602 value of trustworthiness. Some of the issues that may arise here can include the various ways that 603 these existent wearable technologies adopted for the Operator 4.0 uses change efficacy depending on 604 the user. For example, trustworthiness can be severely limited between the technology and the user 605 depending on the physical build of the user. Arm band technologies must ensure that they are 606 adaptable to different skin types, arm diameters and hair density. Likewise, it may be implicitly 607 required that systems recognize the type of user that is currently employing the technology. How a 608 technological intelligent system provides the user an explanation of its goals and actions depends on 609 the user that prompts it, and such can change at any time. These systems must be designed in such a 610 way as to handle dynamic use. How existent wearable technologies employed for the Operator 4.0 611 use can be designed retrospectively to satisfy these issues remains to be seen. 612 5.2. Use Case #2: Augmenting the Workers' Cognitive Capabilities -> Using Simulation-based VR Training 613 Solutions to Enhance Learning 614 Complexity has increased significantly in production environments, thus making traditional 615 training methods outdated. This has made necessary the integration of Modeling & Simulation 616 methods with Virtual Reality to deliver a meaningful environment for personnel training. Prototypes 617 of simulation-based VR training tools for augmenting workers' cognitive capabilities and enhancing 618 the learning speed are presented in Figure 11. In such contexts, at the beginning of the training 619 session, the players are usually given certain directives and rules to respect in order to succeed. 620 Multiple scenarios (even very unlikely ones, such as industrial accidents) may be experienced to test 621 the worker's capability to minimize impulsive behavior and assess the player's improvements over 622 time in terms of compliance to rules and expectations. Figure 12 lays out one potential way to 623 translate design requirements, through norms embodying the conformity value. Of course, various 624 other instantiations in terms of design requirements can better suit such translations. This is, of 625 course, not an exhaustive illustration, as values are interdependent. While on one side, players are 626 Figure 10. Bi-direction hierarchy showcasing how the value of welfare can be translated through norms and into technical design requirements for wearable technology. While values such as welfare have already some background literature (e.g., ergonomics of wearable devices) that may help the designers, some other values are more intricate, for example the value of trustworthiness. Some of the issues that may arise here can include the various ways that these existent wearable technologies adopted for the Operator 4.0 uses change efficacy depending on the user. For example, trustworthiness can be severely limited between the technology and the user depending on the physical build of the user. Arm band technologies must ensure that they are adaptable to different skin types, arm diameters and hair density. Likewise, it may be implicitly required that systems recognize the type of user that is currently employing the technology. How a technological intelligent system provides the user an explanation of its goals and actions depends on the user that prompts it, and such can change at any time. These systems must be designed in such a way as to handle dynamic use. How existent wearable technologies employed for the Operator 4.0 use can be designed retrospectively to satisfy these issues remains to be seen. 5.2. Use Case #2: Augmenting the Workers' Cognitive Capabilities -> Using Simulation-Based VR Training Solutions to Enhance Learn Complexity has increased significantly in production environments, thus making traditional training methods outdat d. This has made necessary the integration of Modeling & Simulation methods with Virtual Reality to deliver a meaningful environment for personnel training. Prototypes of simulation-based V training tools for augmenting wo kers' cognitive capabilities and enhancing the learning speed are prese ted in Figure 11. In such contexts, at the beginning of th training session, pl yers are usually given c rtain directives a d rules to respec in order to succeed. Multiple cenarios (even very unlikely ones, such as industr al accidents) may be exp rienced t test the worker's capability to minimize impulsive behavior and assess the player's improvements over time in terms of compliance to ru es and expectat ons. F gure 12 lays out on potenti l way to t anslate design requirements, through norms embodying the conformity value. Of course, various other i st ntiations in terms of design requirements can better suit such translations. This is, of course, not an exhaustive Appl. Sci. 2020, 10, 4182 18 of 25 illustration, as values are interdependent. While on one side, players are required to conform to the procedure and expectations, on the other side easygoing and uncritical adhesion to norms would minimize the worker's capability to make informed decisions, thus potentially leaving the worker in unknown situations that are risky for health and safety. It emerges therefore the need of a discussion over the correlation of the conformity value with the trustworthiness of VR technology. Technologies that are used or will be used by the workers are shaped by human values but also transform workers' (and people's) identities. Appl. Sci. 2020, 10, x FOR PEER REVIEW 18 of 25 required to conform to the procedure and expectations, on the other side easygoing and uncritical 627 adhesion to norms would minimize the worker's capability to make informed decisions, thus 628 potentially leaving the worker in unknown situations that are risky for health and safety. It emerges 629 therefore the need of a discussion over the correlation of the conformity value with the 630 trustworthiness of VR technology. Technologies that are used or will be used by the workers are 631 shaped by human values but also transform workers' (and people's) identities. 632 633 634 Figure 11. Prototypes of simulation-based VR training tools for augmenting workers' cognitive 635 capabilities. 636 637 Figure 12. Bi-direction hierarchy showcasing how the value of conformity can be extrapolated from 638 technical design requirements through norms for simulation-based VR training tools. 639 Figure 11. Prototypes of simulation-based VR training tools for augmenting workers' cognitive capabilities. ppl. Sci. 2020, 10, x FOR PEER REVIEW 18 of 25 required to conform to the procedure and expectations, on the other side easygoing and uncritical 627 adhesion to norms would minimize the worker's capability to make informed decisions, thus 628 otentially leaving the worker in unknown situations that are risky for healt and safety. It emerges 629 therefore the need of a discussion over the correlation of the conformity value ith the 630 trustworthiness of VR technology. Technologies that are used or will be used by the workers are 631 shaped by human values but also transform workers' (and people's) identities. 632 633 634 Figure 11. Prototypes of simulation-based VR training tools for augmenting workers' cognitive 5 capabilities. 636 637 Figure 12. Bi-direction hierarchy showcasing how the value of conformity can be extrapolated from 638 technical design requirements through norms for simulation-based VR training tools. 639 . t l i ire e ts t ro gh nor s for si ulation-based VR training tools. Appl. Sci. 2020, 10, 4182 19 of 25 5.3. Use Case #3: Augmenting the Workers' Interaction Capabilities -> Using Intelligent Voice-Enabled Digital Assistants at the Factory Voice-enabled digital assistants are drastically changing the way we interact with the CPPS, and prototypes are today available, for example, to support technicians and specialists during maintenance tasks at the factory as showed in Figure 13. Prospectively, every single autonomous and cooperative building block of a CPPS will be able to interact with the human worker via digitally-generated voice responses, thus giving birth to the Social Smart Factory. Digital assistants give workers intuitive and quick access to a plethora of information and knowledge, processed and made available by a ubiquitous intelligence, but concerns exist on how to design 'ethical digital agents' that treat people with consideration and courtesy. Appl. Sci. 2020, 10, x FOR PEER REVIEW 19 of 25 5.3. se ase #3: ug enting the orkers' Interaction Capabilities -> Using Intelligent Voice-Enabled 640 Digit l Assis ants at the Factory 641 oice-enable ssistants are drastically changing the way we interact with the CPPS, and 642 prototypes are today vailab e, for example, to sup ort technicians and ri 643 i at the factory as howed in Figure 13. Prospectively, every single autonomous and 644 cooperative building block of a CPPS will be ab e to int ract with the human worker via digitally-645 generated voice responses, thus giving birth to the Social Sm rt Factory. Digital assistant give 646 workers intuitive and quick ac ess to a plethora of inf rmation nd knowledge, processed and made 647 available by a ubiquitous ntelligence, but concerns xist on how to design 'ethical digital agents' that 648 treat peo l with consideration and courtesy. 649 650 651 Figure 13. Prototypes of voice-enabled digital assistants that augment the worker's interaction 652 capabilities towards the CPPS. 653 Figure 14 tackles the translation of the value of identity for the design of voice-enabled digital 654 assistants, showing how several ethical issues (discrimination of physical-impaired workers, or by 655 language, culture, or current mood) could potentially emerge if value-oriented design requirements 656 are not embedded into 'assistance technology'. One of the issues that can arise is work-related and 657 emotional over-reliance, thus creating a potentially harmful reliance-dependence paradigm where 658 operators are continually encouraged to use these devices for most of their duties. The risk is that a 659 kind of emotional dependence develops, accompanied by overconfidence in the ability of the 660 machine to support one's tasks, even in unexpected situations. Poor design requirements that 661 encompasses all the human values simultaneously can lead to an exacerbation of potentially harmful 662 technical outcomes that then put human welfare in jeopardy given the system reliance on such 663 systems. 664 665 i re 13. r t t es f ice-e a le i ital assista ts t at a e t the worker's interactio capabilities to ards the PPS. Figure 14 tackles the translation of the value of identity for the design of voice-enabled digital assistants, showing how several ethical issues (discrimination of physical-impaired workers, or by language, culture, or current mood) could potentially emerge if value-oriented design requirements are not embedded into 'assistance technology'. One of the issues that can arise is work-related and emotional over-reliance, thus creating a potentially harmful reliance-dependence paradigm where operators are continually encouraged to use these devices for most of their duties. The risk is that a kind of emotional dependence develops, accompanied by overconfidence in the ability of the machine to support one's tasks, even in unexpected situations. Poor design requirements that encompasses all the human values simultaneously can lead to an exacerbation of potentially harmful technical outcomes that then put human welfare in jeopardy given the system reliance on such systems. Appl. Sci. 2020, 10, 4182 20 of 25 Appl. Sci. 2020, 10, x FOR PEER REVIEW 20 of 25 666 Figure 14. Bi-direction hierarchy showcasing how the value of identity can be extrapolated from 667 technical design requirements through norms for voice-enabled digital assistants. 668 6. Conclusions 669 6.1. Contribution of the Study 670 This paper envisages a new revolutionary wave in the industrial sector, Industry 5.0 or Age of 671 Augmentation, where humans and machines cooperate in perfect symbiosis with one another and 672 human capabilities are augmented by a human-friendly technology. Despite arguments that industry 673 is being 'versioned' way too arbitrarily, the human-centered transformation of the smart and 674 automated factories is a completely new paradigm that has been recently anticipated by concepts like 675 'socially sustainable workplaces', 'human-centric factory', and 'Operator 4.0' and is markedly 676 different from the 'techno-centric' Industry 4.0. In this regard, the paper reveals the growing concern 677 in several industrial sectors regarding the ethical issues that arise with technologies that workers 678 currently use or are considering using towards the Factory of the Future. Likewise, they all agreed 679 that all or some of the issues that may arise with these technologies can be responded to and 680 ameliorated by designing these technologies explicitly for human values. 681 This paper argued that a value-oriented and ethical technology engineering in Industry 5.0 is an 682 urgent and sensitive topic. As is true for values in general, which tend to be construed in highly 683 sociocultural nuanced ways, designers require tools to help translate these abstract values, through 684 sociocultural norms (including legal/policy norms) into concrete design requirements. The Value 685 Sensitive Design (VSD) approach has been proposed as a principled framework that design teams of 686 the Factories of the Future can adopt to contribute towards systemic robustness. It illustrates how 687 technologies enabling human–machine symbiosis in the Factory of the Future can be designed to 688 embody elicited human values and shows actionable steps that engineers and designers can take in 689 their design projects. For this reason, this paper is comparatively unique in its approach. Rather than 690 providing a solely speculative application of the VSD methodology to the individual technologies 691 that converge to form the Factory of the Future, we evaluate the applicability of VSD as a pragmatic 692 approach to informing the work of engineers at a specific nexus of practice. In this case, this paper 693 looks at how the adoption of VSD can be used to guide the development of Industry 5.0 solutions. 694 Figure 14. Bi-direction hierarchy showcasing how the value of identity can be extrapolated from technical design requirements through norms for voice-enabled digital assistants. 6. Conclusions 6.1. Contribution of the Study This paper envisages a new revolutionary wave in the industrial sector, Industry 5.0 or Age of Augmentation, where humans and machines cooperate in perfect symbiosis with one another and human capabilities are aug ented by a human-friendly technology. Despite arguments that industry is being 'versioned' way too arbitrarily, the human-centered transformation of the smart and automated factories is a completely new paradigm that has been recently anticipated by concepts like 'socially sustainable workplaces', 'human-centric factory', and 'Operator 4.0' and is markedly different from the 'techno-centric' Industry 4.0. In this regard, the paper reveals the growing concern in several industrial sectors regarding the ethical issues that arise with technologies that workers currently use or are considering using towards the Factory of the Future. Likewise, they all agreed that all or some of the issues that may arise with these technologies can be responded to and ameliorated by designing these technologies explicitly for human values. This paper argued that a value-oriented and ethical technology engineering in Industry 5.0 is an urgent and sensitive topic. As is true for values in general, which tend to be construed in highly sociocultural nuanced ways, designers require tools to help translate these abstract values, through sociocultural norms (including legal/policy norms) into concrete design requirements. The Value Sensitive Design (VSD) approach has been proposed as a principled framework that design teams of the Factories of the Future can adopt to contribute towards systemic robustness. It illustrates how technologies enabling human–machine symbiosis in the Factory of the Future can be designed to embody elicited human values and shows actionable steps that engineers and designers can take in their design projects. For this reason, this paper is comparatively unique in its approach. Rather than providing a solely speculative application of the VSD methodology to the individual technologies that converge to form the Factory of the Future, we evaluate the applicability of VSD as a pragmatic Appl. Sci. 2020, 10, 4182 21 of 25 approach to informing the work of engineers at a specific nexus of practice. In this case, this paper looks at how the adoption of VSD can be used to guide the development of Industry 5.0 solutions. 6.2. Limitations of the Study Still, there are areas of limitations that this paper does not explore but warrant future research. The insights derived from the survey are applicable to a large number of enterprises and sectors but are not exhaustive. Cross-cultural or cross-national comparisons to identify similarities and differences in the specificity of value systems may be needed. Researchers have shown how a value system is not entirely stable but rather dynamically influenced by socio-cultural factors, the changing work world, and transformations in production activities. Therefore, the results of this survey should be intended with reference to enterprises with similar characteristics. Regarding the VSD approach to the human–machine symbiosis in Industry 5.0, the manner in which such values express themselves in a particular culture at a particular point in time can be quite different [35,69]. Investigating relevant stakeholders, determining their values and translating those values into norms and design requirements for any given technological artifact is demanding. Values, as already mentioned, are typically abstract, and conceptualizing how such abstract concepts can be turned into concrete design requirements is understandably laborious. Even the most experienced engineers and design specialists can be stopped in their tracks when trying to determine the inter-relations of values that are in tension and determining how to address such tensions. What must be resisted is the urge towards perfection, something that VSD actively resists. This tendency can hinder learning and meaningful progress [12]. We stipulate this given that the above discussion, until this point, seems idealistic in its attempt to balance so many potential stakeholders and values. Although robustness can be reasonably increased in any project if it is allotted more time and resources, most innovation hubs are constrained by such variables. VSD at its foundation pushes engineers to consider how to achieve the most good given the present constraints and to resist the urge towards the burdensome ideal of perfect innovation. 6.3. Further Research This discussion paves the way for an Industry 5.0 designed for human values. Most of the research efforts focus asymmetrically on the technical and technological aspects of the Factories of the Future while it neglects that a human–machine symbiosis requires profound ethical considerations. The results of our survey showed that 28% of respondent were part of organizations that did not have any formalized code of ethics. This presents both challenges and opportunities for how technologies enabling human–machine symbiosis are designed and used currently within organizations. However, the opportunity to reflect on how to begin codifying the values of the stakeholders both in the organization as well as those who are impacted by that organizations' actions becomes apparent. Future research should look at the salience of existent standards and codes of practice towards O4.0 technologies as well as practices for developing new standards and guidelines alongside the ethical development and use of those technologies. Co-construction provides the most salient mode of policy innovation and VSD is similarly apt to providing a means by which both technologies and their respective governance and policy structures can be co-constructed [68]. Future work will also be designed to assess whether there are significant differences between organizational members and leaders, comparing for instance, the values of pairs of supervisors and subordinates as well as significant differences among enterprises depending on their size. Further elicitation methods will be used to ask for additional value considerations, organizational motivations, methods of training and dissemination, reward structures, and economic incentives with reference to ethics at the workplace. Ethics training for executives, managers and employees may be required in the next future as well as the need for a 24 h hot line for ethical situational advice that may aid workers or employers in the case of ambiguous situations. Future works should clarify when automation starts decreasing and affecting the workers' quality of life ('over-automation') and understand who decides that developing Appl. Sci. 2020, 10, 4182 22 of 25 a certain kind of new technology is 'safe' for the workforce. From our review, we noticed that the main proposals fostering the adoption of techno ethics considerations are often related to the idea of charters to be signed ('Hippocratic oath'). However, the legal system is having trouble in catching up with the increasing speed of technological advances. In this sense, the iterative VSD approach may dramatically help in the identification of ethical rules and frameworks that technologies for a human–machine symbiosis in the Factory of the Future must be designed for. Supplementary Materials: The following are available online at http://www.mdpi.com/2076-3417/10/12/4182/s1, Video S1: Voice-enabled Digital Assistant. Author Contributions: The authors have contributed equally to the paper. All authors have read and agreed to the published version of the manuscript. Funding: The research work described in this paper is part of the activities carried out in the context of the SO4SIMS project (Smart Operators 4.0 based on Simulation for Industry and Manufacturing Systems) funded by the Italian Ministry of Education, Universities and Research MIUR (Project PRIN – 2017FW8BB4). Acknowledgments: The use cases and prototypes showed in the survey build upon solutions and projects developed by CAL-TEK S.r.l., a Spin-off company of the University of Calabria, Italy and carried out at the Modeling & Simulation Center – Laboratory of Enterprise Solutions (MSC-LES) at the Department of Mechanical, Energy and Management Engineering of the University of Calabria, such as the W-Artemys project, financed by INAIL (Italian National Institute for Insurance against Accidents at Work). The authors also acknowledge all the people who participated to the survey. Conflicts of Interest: The authors declare no conflict of interest. References 1. Monostori, L.; Kádár, B.; Bauernhansl, T.; Kondoh, S.; Kumara, S.; Reinhart, G.; Sauer, O.; Schuh, G.; Sihn, W.; Ueda, K. Cyber-physical systems in manufacturing. CIRP Ann. 2016, 65, 621–641. [CrossRef] 2. Krugh, M.; Mears, L. A complementary Cyber-Human Systems framework for Industry 4.0 Cyber-Physical Systems. Manuf. Lett. 2018, 15, 89–92. [CrossRef] 3. Hancock, P.A.; Jagacinski, R.J.; Parasuraman, R.; Wickens, C.D.; Wilson, G.F.; Kaber, D.B. Human-automation interaction research: Past, present, and future. Ergon. Des. 2013, 21, 9–14. [CrossRef] 4. Cohen, Y.; Golan, M.; Singer, G.; Faccio, M. Workstation-Operator Interaction in 4.0 Era: WOI 4.0. IFAC-PapersOnLine 2018, 51, 399–404. [CrossRef] 5. Özdemir, V.; Hekim, N. Birth of Industry 5.0: Making Sense of Big Data with Artificial Intelligence, "The Internet of Things" and Next-Generation Technology Policy. Omics A J. Integr. Biol. 2018, 22, 65–76. [CrossRef] 6. European Economic and Social Committee. Industry 5.0. Available online: https://www.eesc.europa.eu/en/ agenda/our-events/events/industry-50 (accessed on 7 April 2020). 7. Philbeck, T.; Davis, N.; Engtoft Larsen, A.M. Values, Ethics and Innovation-Rethinking Technological Development in the Fourth Industrial Revolution; World Economic Forum: Cologny, Switzerland, 2018. 8. de Saille, S. Innovating innovation policy: The emergence of 'Responsible Research and Innovation'. J. Responsible Innov. 2015, 2, 152–168. [CrossRef] 9. United Nations. The Age of Digital Interdependence-Digital cooperation: Report of the UN SecretaryGeneral's High-level Panel on Digital Cooperation. Available online: https://digitalcooperation.org/wpcontent/uploads/2019/06/DigitalCooperation-report-for-web.pdf (accessed on 25 January 2020). 10. van de Poel, I. Design for Values in Engineering. In Handbook of Ethics, Values, and Technological Design: Sources, Theory, Values and Application Domains; van den Hoven, J., Vermaas, P.E., van de Poel, I., Eds.; Springer: Dordrecht, The Netherlands, 2014; pp. 1–20. ISBN 978-94-007-6994-6. 11. Kiran, A.H.; Oudshoorn, N.; Verbeek, P.P. Beyond checklists: Toward an ethical-constructive technology assessment. J. Responsible Innov. 2015, 2, 5–19. [CrossRef] 12. Friedman, B.; Hendry, D.G. Value Sensitive Design: Shaping Technology with Moral Imagination; Mit Press: Cambridge, MA, USA, 2019; ISBN 9780262039536. 13. Kaasinen, E.; Schmalfuss, F.; Özturk, C.; Aromaa, S.; Boubekeur, M.; Heilala, J.; Heikkilä, P.; Kuula, T.; Liinasuo, M.; Mach, S.; et al. Empowering and engaging industrial workers with Operator 4.0 solutions. Comput. Ind. Eng. 2020, 139, 105678. [CrossRef] Appl. Sci. 2020, 10, 4182 23 of 25 14. Calabrese, A.; Levialdi Ghiron, N.; Tiburzi, L. 'Evolutions' and 'revolutions' in manufacturers' implementation of industry 4.0: A literature review, a multiple case study, and a conceptual framework. Prod. Plan. Control 2020, 1–15. [CrossRef] 15. Müller, J.M.; Buliga, O.; Voigt, K.I. Fortune favors the prepared: How SMEs approach business model innovations in Industry 4.0. Technol. Forecast. Soc. Chang. 2018, 132, 2–17. [CrossRef] 16. Romero, D.; Bernus, P.; Noran, O.; Stahre, J.; Fast-Berglund, Å. The operator 4.0: Human cyber-physical systems & adaptive automation towards human-automation symbiosis work systems. In Proceedings of the APMS 2016: Advances in Production Management Systems. Initiatives for a Sustainable World, Iguassu Falls, Brazil, 3–7 September 2016; Nääs, I., Vendrametto, O., Reis, J.M., Gonçalves, R.F., Silva, M.T., von Cieminski, G., Kiritsis, D., Eds.; Springer: Berlin/Heidelberg, Germany, 2017; Volume 488, pp. 677–686. 17. Guerin, C.; Rauffet, P.; Chauvin, C.; Martin, E. Toward production operator 4.0: Modelling Human-Machine Cooperation in Industry 4.0 with Cognitive Work Analysis. IFAC-PapersOnLine 2019, 52, 73–78. [CrossRef] 18. Salaken, S.M.; Nahavandi, S.; McGinn, C.; Hossny, M.; Kelly, K.; Abobakr, A.; Nahavandi, D.; Iskander, J. Development of a cloud-based computational framework for an empathetic robot. In Proceedings of the 11th International Conference on Computer and Automation Engineering, Perth, Australia, 23–25 Feburary 2019; pp. 102–108. 19. Zolotová, I.; Papcun, P.; Kajáti, E.; Miškuf, M.; Mocnej, J. Smart and cognitive solutions for Operator 4.0: Laboratory H-CPPS case studies. Comput. Ind. Eng. 2018, 139, 105471. [CrossRef] 20. Romero, D.; Stahre, J.; Wuest, T.; Noran, O.; Bernus, P.; Fast-Berglund, Å.; Gorecky, D. Towards an Operator 4.0 Typology: A Human-Centric Perspective on the Fourth Industrial Revolution Technologies. In Proceedings of the CIE 2016: 46th International Conferences on Computers and Industrial Engineering, Tianjin, China, 29–31 October 2016. 21. Romero, D.; Mattsson, S.; Fast-Berglund, Å.; Wuest, T.; Gorecky, D.; Stahre, J. Digitalizing occupational health, safety and productivity for the operator 4.0. In Proceedings of the IFIP Advances in Information and Communication Technology, Seoul, Korea, 23–26 August 2018; Springer: New York, NY, USA, 2018; Volume 536, pp. 473–481. 22. Papcun, P.; Kajati, E.; Koziorek, J. Human machine interface in concept of industry 4.0. In Proceedings of the DISA 2018-IEEE World Symposium on Digital Intelligence for Systems and Machines (DISA), Kosice, Slovakia, 23–25 August 2018; Institute of Electrical and Electronics Engineers Inc.: Piscataway, NJ, USA, 2018; pp. 289–296. 23. Taylor, M.P.; Boxall, P.; Chen, J.J.J.; Xu, X.; Liew, A.; Adeniji, A. Operator 4.0 or Maker 1.0? Exploring the implications of Industrie 4.0 for innovation, safety and quality of work in small economies and enterprises. Comput. Ind. Eng. 2020, 139. [CrossRef] 24. Koch, P.J.; van Amstel, M.K.; Dębska, P.; Thormann, M.A.; Tetzlaff, A.J.; Bøgh, S.; Chrysostomou, D. A Skill-based Robot Co-worker for Industrial Maintenance Tasks. Procedia Manuf. 2017, 11, 83–90. [CrossRef] 25. Rahman, M.H.; Saad, M.; Kenné, J.P.; Archambault, P.S. Control of an exoskeleton robot arm with sliding mode exponential reaching law. Int. J. Control. Autom. Syst. 2013, 11, 92–104. [CrossRef] 26. Trentesaux, D.; Rault, R. Designing Ethical Cyber-Physical Industrial Systems. IFAC-PapersOnLine 2017, 50, 14934–14939. [CrossRef] 27. Bednar, P.M.; Welch, C. Socio-Technical Perspectives on Smart Working: Creating Meaningful and Sustainable Systems. Inf. Syst. Front. 2019, 1–18. [CrossRef] 28. Pacaux-Lemoine, M.P.; Trentesaux, D. Ethical Risks of Human-Machine Symbiosis in Industry 4.0: Insights from the Human-Machine Cooperation Approach. IFAC-PapersOnLine 2019, 52, 19–24. [CrossRef] 29. Demir, K.A.; Döven, G.; Sezen, B. Industry 5.0 and Human-Robot Co-working. Procedia Comput. Sci. 2019, 158, 688–695. [CrossRef] 30. Welfare, K.S.; Hallowell, M.R.; Shah, J.A.; Riek, L.D. Consider the Human Work Experience When Integrating Robotics in the Workplace. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, Daegu, Korea, 11–14 March 2019; pp. 75–84. 31. Grässler, I.; Pöhler, A.; Pottebaum, J. Creation of a Learning Factory for Cyber Physical Production Systems. Procedia CIRP 2016, 54, 107–112. [CrossRef] 32. Dworschak, B.; Zaiser, H. Competences for cyber-physical systems in manufacturing-First findings and scenarios. Procedia CIRP 2014, 25, 345–350. [CrossRef] Appl. Sci. 2020, 10, 4182 24 of 25 33. van de Poel, I. Conflicting Values in Design. In Handbook of Ethics, Values, and Technological Design: Sources, Theory, Values and Application Domains; van den Hoven, J., Vermaas, P.E., van de Poel, I., Eds.; Springer: Dordrecht, The Netherlands, 2014; pp. 1–23. ISBN 978-94-007-6994-6. 34. Horváth, D.; Szabó, R.Z. Driving forces and barriers of Industry 4.0: Do multinational and small and medium-sized companies have equal opportunities? Technol. Forecast. Soc. Chang. 2019, 146, 119–132. [CrossRef] 35. Friedman, B.; Hendry, D.G.; Borning, A. A Survey of Value Sensitive Design Methods. Found. Trends®Hum. –Comput. Interact. 2017, 11, 63–125. [CrossRef] 36. Avallone, F.; Farnese, M.L.; Pepe, S.; Vecchione, M. The Work Values Questionnaire (WVQ): Revisiting Schwartz's Portrait Values Questionnaire (PVQ) for work contexts. BPA-Appl. Psychol. Bull. 2010, 261, 59–76. 37. Friedman, B.; Kahn, P.H.; Borning, A. Value Sensitive Design and Information Systems. In The Handbook of Information and Computer Ethics; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2008; pp. 69–101. ISBN 9780471799597. 38. Lindemann, N. What's the Average Survey Response Rate? [2019 Benchmark]. Available online: https://surveyanyplace.com/average-survey-response-rate/#:~{}:text=A%20low%20response%20rate% 20can,the%20participants%20regarding%20the%20outcome.&text=However%2C%20recent%20studies% 20have%20shown,near%2060%20or%2070%25 (accessed on 25 January 2020). 39. Ruzic, L.; Sanfod, J.A. Universal Design Mobile Interface Guidelines (UDMIG) for an Aging Population. In Mobile e-Health; Springer: Berlin/Heidelberg, Germany, 2017; pp. 17–37. 40. Newell, A.F.; Gregor, P.; Morgan, M.; Pullin, G.; Macaulay, C. User-Sensitive Inclusive Design. Univers. Access Inf. Soc. 2011, 10, 235–243. [CrossRef] 41. Bhamra, T.; Lofthouse, V. Design for Sustainability: A Practical Approach; Routledge: Abingdon, UK, 2016; ISBN 1317152352. 42. Bødker, K.; Kensing, F.; Simonsen, J. Participatory IT Design: Designing for Business and Workplace Realities; MIT Press: Cambridge, MA, USA, 2009; ISBN 0262261790. 43. Davis, J.; Nathan, L.P. Handbook of Ethics, Values, and Technological Design: Sources, Theory, Values and Application Domains; van den Hoven, J., Vermaas, P.E., van de Poel, I., Eds.; Springer: Berlin/Heidelberg, Germany, 2015; pp. 12–40. ISBN 9789400769700. 44. Umbrello, S. Beneficial Artificial Intelligence Coordination by Means of a Value Sensitive Design Approach. Big Data Cogn. Comput. 2019, 3, 5. [CrossRef] 45. Umbrello, S. Atomically Precise Manufacturing and Responsible Innovation: A Value Sensitive Design Approach to Explorative Nanophilosophy. Int. J. Technoethics 2019, 10, 1–21. [CrossRef] 46. Winkler, T.; Spiekermann, S. Twenty years of value sensitive design: A review of methodological practices in VSD projects. Ethics Inf. Technol. 2018, 1–5. [CrossRef] 47. van den Hoven, J.; Manders-Huits, N. Value-Sensitive Design. In A Companion to the Philosophy of Technology; Wiley-Blackwell: Hoboken, NJ, USA, 2009; pp. 477–480. ISBN 9781444310795. 48. Umbrello, S. The moral psychology of value sensitive design: The methodological issues of moral intuitions for responsible innovation. J. Responsible Innov. 2018, 5, 186–200. [CrossRef] 49. Borning, A.; Muller, M. Next steps for value sensitive design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, TX, USA, 5–10 May 2012; p. 1125. [CrossRef] 50. Oosterlaken, I. Applying Value Sensitive Design (VSD) to Wind Turbines and Wind Parks: An Exploration. Sci. Eng. Ethics 2015, 21, 359–379. [CrossRef] 51. Mouter, N.; de Geest, A.; Doorn, N. A values-based approach to energy controversies: Value-sensitive design applied to the Groningen gas controversy in the Netherlands. Energy Policy 2018, 122, 639–648. [CrossRef] 52. Woelfer, J.P.; Iverson, A.; Hendry, D.G.; Friedman, B.; Gill, B.T. Improving the Safety of Homeless Young People with Mobile Phones: Values, Form and Function. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada, 7 May 2011; ACM: New York, NY, USA, 2011; pp. 1707–1716. 53. van den Hoven, J. Architecture and Value-Sensitive Design. In Ethics, Design and Planning of the Built Environment; Basta, C., Moroni, S., Eds.; Springer Science & Business Media: Berlin, Germany, 2013; p. 224. ISBN 9400752466. Appl. Sci. 2020, 10, 4182 25 of 25 54. Friedman, B.; Kahn, P.H., Jr. New Directions: A Value-sensitive Design Approach to Augmented Reality. In Proceedings of the DARE 2000 on Designing Augmented Reality Environments, Elsinore, Denmark, 1 April 2000; ACM: New York, NY, USA, 2000; pp. 163–164. 55. Timmermans, J.; Zhao, Y.; van den Hoven, J. Ethics and Nanopharmacy: Value Sensitive Design of New Drugs. Nanoethics 2011, 5, 269–283. [CrossRef] [PubMed] 56. Umbrello, S.; De Bellis, A.F. A Value-Sensitive Design Approach to Intelligent Agents. In Artificial Intelligence Safety and Security; Yampolskiy, R.V., Ed.; CRC Press: Boca Raton, FL, USA, 2018; pp. 395–410. ISBN 9780815369820. 57. Calvert, S.C.; Mecacci, G.; Heikoop, D.D.; de Sio, F.S. Full platoon control in Truck Platooning: A Meaningful Human Control perspective. In Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018; pp. 3320–3326. 58. Super, D.E. Work Values Inventory; Houghton Mifflin: Boston, MA, USA, 1968. 59. Super, D.E.; Sverko, B. Life roles, values, and careers: International findings of the Work Importance Study. In Life Roles, Values, and Careers: International Findings of the Work Importance Study; Jossey-Bass: San Francisco, CA, USA, 1995. 60. Schwartz, S.H. A Proposal for Measuring Value Orientations across Nations. In Core ESS Questionnaire; ESS: London, UK, 2003; ISBN 047166782X. 61. Brey, P.A.E. Anticipatory Ethics for Emerging Technologies. Nanoethics 2012, 6, 1–13. [CrossRef] 62. Vallor, S. Technology and the Virtues: A Philosophical Guide for a Future Worth Wanting; Oxford University Press: New York, NY, USA, 2016; ISBN 9780190498511. 63. La Fors, K.; Custers, B.; Keymolen, E. Reassessing values for emerging big data technologies: Integrating design-based and application-based approaches. Ethics Inf. Technol. 2019, 21, 209–226. [CrossRef] 64. Yoo, D. Stakeholder Tokens: A constructive method for value sensitive design stakeholder analysis. In Proceedings of the 2017 ACM Conference Companion Publication on Designing Interactive Systems, Edinburgh, UK, 6–10 June 2017; ACM: New York, NY, USA, 2017; pp. 280–284. 65. Freeman, B.; Borning, A. YP and urban simulation: Applying an agile programming methodology in a politically tempestuous domain. In Proceedings of the Agile Development Conference, Salt Lake City, UT, USA, 25–28 June 2003; pp. 2–11. 66. Nathan, L.P.; Klasnja, P.V.; Friedman, B. Value Scenarios: A Technique for Envisioning Systemic Effects of New Technologies. In Proceedings of the CHI 07 Extended Abstracts on Human Factors in Computing Systems, San Jose, CA, USA, 28 April 2007; ACM: New York, NY, USA, 2007; pp. 2585–2590. 67. Friedman, B.; Kahn, P.H., Jr.; Borning, A.; Huldtgren, A. Value Sensitive Design and Information Systems. In Early Engagement and New Technologies: Opening up the Laboratory; Doorn, N., Schuurbiers, D., van de Poel, I., Gorman, M.E., Eds.; Springer: Dordrecht, The Netherlands, 2013; pp. 55–95. ISBN 978-94-007-7844-3. 68. van de Poel, I. Translating Values into Design Requirements BT. In Philosophy and Engineering: Reflections on Practice, Principles and Process; Michelfelder, D.P., McCarthy, N., Goldberg, D.E., Eds.; Springer: Dordrecht, The Netherlands, 2013; pp. 253–266. ISBN 978-94-007-7762-0. 69. Umbrello, S. Imaginative Value Sensitive Design: Using Moral Imagination Theory to Inform Responsible Technology Design. Sci. Eng. Ethics 2019, 25, 1–21. [CrossRef] © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).