Mouse Facial expressions
109056507
(224) An Open Source Deep Neural Network to Assess Spontaneous Pain in Freely Moving Mice using Facial Expressions
There is a dire need for new non opioid analgesics. However, efforts to develop new pain medicines have thus far met with limited success. This failure is partly due to an overreliance on evoked pain measures in preclinical models. Most preclinical models do not measure spontaneous pain the main symptom of chronic pain in humans. Here we show our progress towards developing and validating a fully automated and user friendly spontaneous pain measurement tool by training a machine learning model to detect and quantify facial grimacing in mice. The original Mouse Grimace Scale (MGS) was developed to train researchers to detect pain in freely moving mice by quantifying characteristic facial expressions, or "grimacing" To increase the rate of adoption for this spontaneous pain measure, we recently adapted and trained a general object classification convolutional neural network (Google's Inception v1.0TM) to detect and classify the presence or absence of grimace in still images of freely moving white mice. We call this recently published model the automated Mouse Grimace Scale, aMGS. Here we show our continuing efforts to further improve our model using new convolutional neural architecture to detect grimacing in mice of various coat colors and across different laboratory settings. Unlike our first aMGS program, our newest model can track mouse faces of various colors using continuous video input, rather than relying on still images to determine pain status. We also show the results of our latest validation experiments, including grimace assessment in mice following laparotomy, a common post surgical pain assay. The rapidity and utility of our proposed automated pain classifier has the potential to greatly increase the adoption of the mouse grimace as a reproducible and clinically relevant pain measure, and allow researchers to objectively and rapidly assess tonic and chronic pain states in a variety of mouse models using a measure of spontaneous pain.
The Journal of Pain
Alexander H Tuttle G Wu Mark J Zylka
59307119
Robo1 and 2 Repellent Receptors Cooperate to Guide Facial Neuron Cell Migration and Axon Projections in the Embryonic Mouse Hindbrain
The facial nerve is necessary for our ability to eat, speak, and make facial expressions. Both the axons and cell bodies of the facial nerve undergo a complex embryonic developmental pattern involving migration of the cell bodies caudally and tangentially through rhombomeres, and simultaneously the axons projecting to exit the hindbrain to form the facial nerve. Our goal in this study was to test the functions of the chemorepulsive receptors Robo1 and Robo2 in facial neuron migration and axon projection by analyzing genetically marked motor neurons in double mutant mouse embryos through the migration time course, E10.0 E13.5. In Robo1/2 double mutants, axon projection and cell body migration errors were more severe than in single mutants. Most axons did not make it to their motor exit point, and instead projected into and longitudinally within the floor plate. Surprisingly, some facial neurons had multiple axons exiting and projecting into the floor plate. At the same time, a subset of mutant facial cell bodies failed to migrate caudally, and instead either streamed dorsally toward the exit point or shifted into the floor plate. We conclude that Robo1 and Robo2 have redundant functions to guide multiple aspects of the complex cell migration of the facial nucleus, as well as regulating axon trajectories and suppressing formation of ectopic axons.
Neuroscience
Hannah N Gruner Minkyung Kim Grant S Mastick
214763289
Facial expressions of emotion states and their neuronal correlates in mice
How to read the face of a mouse The neuroscientific investigation of emotions is hindered by a lack of rapid and precise readouts of emotion states in model organisms. Dolensek et al. identified facial expressions as innate and sensitive reflections of the internal emotion state in mice (see the Perspective by Girard and Bellone) Mouse facial expressions evoked by diverse stimuli could be classified into emotionlike categories, similar to basic emotions in humans. Machine learning algorithms categorized mouse facial expressions objectively and quantitatively at millisecond time scales. Intensity, value, and persistence of subjective emotion states could thus be decoded in individual animals. Combining facial expression analysis with two photon calcium imaging allowed the identification of single neurons whose activity closely correlated with specific facial expressions in the insular cortex, a brain region implicated in affective experiences in humans. Science, this issue p. 89; see also p. 33 In mice, facial expressions can be used to infer an animal's current emotions and thereby allow mechanistic study of basic emotions. Understanding the neurobiological underpinnings of emotion relies on objective readouts of the emotional state of an individual, which remains a major challenge especially in animal models. We found that mice exhibit stereotyped facial expressions in response to emotionally salient events, as well as upon targeted manipulations in emotion relevant neuronal circuits. Facial expressions were classified into distinct categories using machine learning and reflected the changing intrinsic value of the same sensory stimulus encountered under different homeostatic or affective conditions. Facial expressions revealed emotion features such as intensity, valence, and persistence. Two photon imaging uncovered insular cortical neuron activity that correlated with specific facial expressions and may encode distinct emotions. Facial expressions thus provide a means to infer emotion states and their neuronal correlates in mice.
Science
Nejc Dolensek Daniel A Gehrlach Alexandra S Klein Nadine Gogolla
215794496
Towards a fully automated surveillance of well being status in laboratory mice using deep learning: Starting with facial expression analysis
Assessing the well being of an animal is hindered by the limitations of efficient communication between humans and animals. Instead of direct communication, a variety of parameters are employed to evaluate the well being of an animal. Especially in the field of biomedical research, scientifically sound tools to assess pain, suffering, and distress for experimental animals are highly demanded due to ethical and legal reasons. For mice, the most commonly used laboratory animals, a valuable tool is the Mouse Grimace Scale (MGS) a coding system for facial expressions of pain in mice. We aim to develop a fully automated system for the surveillance of post surgical and post anesthetic effects in mice. Our work introduces a semi automated pipeline as a first step towards this goal. A new data set of images of black furred laboratory mice that were moving freely is used and provided. Images were obtained after anesthesia (with isoflurane or ketamine/xylazine combination) and surgery (castration) We deploy two pre trained state of the art deep convolutional neural network (CNN) architectures (ResNet50 and InceptionV3) and compare to a third CNN architecture without pre training. Depending on the particular treatment, we achieve an accuracy of up to 99% for the recognition of the absence or presence of post surgical and/or post anesthetic effects on the facial expression.
PloS one
Niek Andresen Manuel Wollhaf Katharina Hohlbaum Lars Lewejohann O Hellwich Christa Thone-Reineke Vitaly Belik
91585605
Towards a fully automated surveillance of well being status in laboratory mice using deep learning
Assessing the well being of an animal is hindered by the limitations of efficient communication between humans and animals. Instead of direct communication, a variety of behavioral, biochemical, physiological, and physical parameters are employed to evaluate the well being of an animal. Especially in the field of biomedical research, scientifically sound tools to assess pain, suffering, and distress for experimental animals are highly demanded due to ethical and legal reasons. For mice, the most commonly used laboratory animals, a valuable tool is the Mouse Grimace Scale (MGS) a coding system for facial expressions of pain in mice which has been shown to be accurate and reliable. Currently, MGS scoring is very time and effort consuming as it is manually performed by humans being thoroughly trained in using this method. Therefore, we aim to develop a fully automated system for the surveillance of well being in mice. Our work introduces a semi automated pipeline as a first step towards this goal. We use and provide a new data set of images of black furred laboratory mice that were moving freely, thus the images contain natural variation with regard to perspective and background. The analysis of this data set is therefore more challenging but reflects realistic conditions as it would be obtainable without human intervention. Images were obtained after anesthesia (with isoflurane or ketamine/xylazine combination) and surgery (castration) We deploy two pre trained state of the art deep convolutional neural network (CNN) architectures (ResNet50 and InceptionV3) and compare to a third CNN architecture without pre training. Depending on the particular treatment, we achieve an accuracy of up to 99% for binary "pain""no pain" classification. Author summary In the field of animal research, it is crucial to assess the well being of an animal. For mice, the most commonly used laboratory animals, there is a variety of indicators for well being. Especially the facial expression of a mouse can give us important information on its well being state. However, currently the surveillance of well being can only be ensured if a human is present. Therefore, we developed a first approach towards a fully automated surveillance of the well being status of a mouse. We trained neural networks on face images of black furred mice, which were either untreated or underwent anesthesia or surgery, to distinguish between an impaired and unimpaired well being state. Our systems successfully learnt to assess whether the well being of a mouse was impaired and, depending on the particular treatment, its decision was correct in up to 99% A tool that visualizes the features used for the decision making process indicated that the decision was mainly based on the facial expressions of a mouse.
Niek Andresen Manuel Wollhaf Katharina Hohlbaum Lars Lewejohann Olaf Hellwich Christa Thone-Reineke Vitaly Belik
195893308
An empirical evaluation of a hands free computer interaction for users with motor disabilities
Standard computer input devices such as a mouse or a keyboard are not well suited to the needs of users with severe motor disabilities in their interaction with standard computer interfaces. The emergence of contemporary human computer interfaces has allowed for the development of innovative solutions for hands free Human Computer Interaction (HCI) which can improve the quality and accessibility of Information and Communication Technology (ICT) for motor impaired users. The objectives of this study were to design, develop and evaluate a solution for a hands free HCI, based on the Emotiv EPOC+ device, which, among other capabilities, also enables controlling the computer with facial expressions and motion sensors. Ten non disabled adults and eight adults with a severe motor disability participated in an experiment to evaluate the proposed HCI solution. Eighteen users completed six experimental tasks successfully using both their existing computer use approach as well as the proposed hands free computer use approach. The times necessary to complete the tasks were measured and analyzed, along with users' subjective observations about the difficulty level of both computer use approaches. Users' perceptions about the new hands free computer use approach were assessed as well. Although there were no significant differences in both user types regarding the difficulty level in completing the tasks, disabled users solved the tasks with less effort. Positive perceptions about perceived usefulness, ease of use, appropriateness of the technology, and satisfaction with the proposed solution for touchless interaction were assessed for both user types. Scores were significantly higher for disabled users in the case of measuring the perceived usefulness, perceived ease of use and satisfaction with the solution. This study showed that users with severe motor difficulties find new HCI less challenging compared to their existing computer use approach than the non disabled who use standard HCI. When compared with non disabled users, the disabled ones can be equally effective when confronted with a new HCI technology. Future work is needed to improve the proposed solution and to analyze the impact of different factors on users with motor disabilities and their adoption of an innovative technology for touchless interaction with a computer.
J. Biomed. Informatics
Bostjan Sumak Matic Spindler Mojca Debeljak Marjan Hericko Maja Pusnik
204863279
Recognizing Multidimensional Engagement of E Learners Based on Multi Channel Data in E Learning Environment
"Lack of supervision" is a particularly challenging problem in E learning or distance learning environments. A wide range of research efforts and technologies have been explored to alleviate its impact by monitoring students' engagement, such as emotion or learning behaviors. However, the current research still lacks multi dimensional computational measures for analyzing learner's engagement from the interactions that occur in digital learning environment. In this paper, we propose an integrated framework to identify learning engagement from three facets: affect, behavior and cognitive state, which are conveyed by learner's facial expressions, eye movement behaviors and the overall performance during short video learning session. To recognize the three states of learners, three channel data is recorded: 1) video/image sequence captured by camera; 2) eye movement information from a non intrusive and cost effective eye tracker; and 3) click stream data from mouse. Based on these modalities, a multi channel data fusion strategy is designed that concatenates time series features of three channels in the same time segment to predict course learning performance. We also presented a new method to make the self reported annotations more reliable without using external observers' verification. To validate the approach and methods, 46 participants were invited to attend a representative on line course that consists of short videos in our designed learning environment. The results demonstrated the effectiveness of the proposed framework and methods in monitoring learning engagement. More importantly, a prototype system was developed to detect learner's emotional and eye behavioral engagement in real time as well as predict the learning performance of learners after they had completed each short video course.
IEEE Access
Jia Yue Feng Tian Kuo-Min Chao Nazaraf Shah Longzhuang Li Yuanchun Chen Qinghua Zheng
214761400
Critical Success Factors of Analysing User Emotions to Improve the Usability of Systems
Emotions of a person depend on several factors like personality, relationships, mental health, and among them, context also plays a major role. The acceptability and usability of a system can be enhanced when the user emotions are considered based on the context. Most of the past research has been carried out related to the detection of emotions using facial expressions, gaze direction, heart rate analysis, Electroencephalogram (EEG) signals, text analytics, and keystroke dynamics along with mouse movements. This paper is a literature review which summarizes the previous research work in order to identify the critical success factors of the existing approaches when detecting user emotions through facial expressions and text data along with the context. Finally, as a conclusion, considering the advantages and drawbacks of the existing applications, the novel approaches that need to be followed to analyse user emotions to improve the usability of systems are proposed.
2019 Twelfth International Conference on Ubi Media Computing (Ubi Media)
Wasura D Wattearachchi Enosha Hettiarachchi K Priyantha Hewagamage
19525052
Eyemotion: Classifying Facial Expressions in VR Using Eye Tracking Cameras
One of the main challenges of social interaction in virtual reality settings is that head mounted displays occlude a large portion of the face, blocking facial expressions and thereby restricting social engagement cues among users. We present an algorithm to automatically infer expressions by analyzing only a partially occluded face while the user is engaged in a virtual reality experience. Specifically, we show that images of the user's eyes captured from an IR gaze tracking camera within a VR headset are sufficient to infer a subset of facial expressions without the use of any fixed external camera. Using these inferences, we can generate dynamic avatars in real time which function as an expressive surrogate for the user. We propose a novel data collection pipeline as well as a novel approach for increasing CNN accuracy via personalization. Our results show a mean accuracy of 74% (F1 of 0.73) among 5 'emotive' expressions and a mean accuracy of 70% (F1 of 0.68) among 10 distinct facial action units, outperforming human raters.
2019 IEEE Winter Conference on Applications of Computer Vision (WACV)
Steven Hickson Nick Dufour Avneesh Sud Vivek Kwatra Irfan Essa
150798263
Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements
It is commonly assumed that a person's emotional state can be readily inferred from his or her facial movements, typically called emotional expressions or facial expressions. This assumption influences legal judgments, policy decisions, national security protocols, and educational practices; guides the diagnosis and treatment of psychiatric illness, as well as the development of commercial applications; and pervades everyday social interactions as well as research in other scientific fields such as artificial intelligence, neuroscience, and computer vision. In this article, we survey examples of this widespread assumption, which we refer to as the common view, and we then examine the scientific evidence that tests this view, focusing on the six most popular emotion categories used by consumers of emotion research: anger, disgust, fear, happiness, sadness, and surprise. The available scientific evidence suggests that people do sometimes smile when happy, frown when sad, scowl when angry, and so on, as proposed by the common view, more than what would be expected by chance. Yet how people communicate anger, disgust, fear, happiness, sadness, and surprise varies substantially across cultures, situations, and even across people within a single situation. Furthermore, similar configurations of facial movements variably express instances of more than one emotion category. In fact, a given configuration of facial movements, such as a scowl, often communicates something other than an emotional state. Scientists agree that facial movements convey a range of information and are important for social communication, emotional or otherwise. But our review suggests an urgent need for research that examines how people actually move their faces to express emotions and other social information in the variety of contexts that make up everyday life, as well as careful study of the mechanisms by which people perceive instances of emotion in one another. We make specific research recommendations that will yield a more valid picture of how people move their faces to express emotions and how they infer emotional meaning from facial movements in situations of everyday life. This research is crucial to provide consumers of emotion research with the translational information they require.
Psychological science in the public interest a journal of the American Psychological Society
Lisa Feldman Barrett Ralph Adolphs Stacy Marsella Aleix M Martinez Seth D Pollak