virtual reality and machine learning
221283008
Subjective QoE of 360 Degree Virtual Reality Videos and Machine Learning Predictions
360 degree video provides an immersive experience to end users through Virtual Reality (VR) Head Mounted Displays (HMDs) However, it is not trivial to understand the Quality of Experience (QoE) of 360 degree video since user experience is influenced by various factors that affect QoE when watching a 360 degree video in VR. This manuscript presents a machine learning based QoE prediction of 360 degree video in VR, considering the two key QoE aspects: perceptual quality and cybersickness. In addition, we proposed two new QoE affecting factors: user's familiarity with VR and user's interest in 360 degree video for the QoE evaluation. To aim this, we first conduct a subjective experiment on 96 video samples and collect datasets from 29 users for perceptual quality and cybersickness. We design a new Logistic Regression (LR) based model for QoE prediction in terms of perceptual quality. The prediction accuracy of the proposed model is compared against well known supervised machine learning algorithms such as k Nearest Neighbors (kNN) Support Vector Machine (SVM) and Decision Tree (DT) with respect to accuracy rate, recall, f1 score, precision, and mean absolute error (MAE) LR performs well with 86% accuracy, which is in close agreement with subjective opinion. The prediction accuracy of the proposed model is then compared with existing QoE models in terms of perceptual quality. Finally, we build a Neural Network based model for the QoE prediction in terms of cybersickness. The proposed model performs well against the state of the art QoE prediction methods in terms of cybersickness.
IEEE Access
Muhammad Shahid Anwar Jing Wang Wahab Khan Asad Ullah Sadique Ahmad Zesong Fei
229308503
Virtual reality sickness detection: an approach based on physiological signals and machine learning
Virtual Reality (VR) is spreading to the general public but still has a major issue: VR sickness. To take it into consideration and minimize its occurrence, evaluation methods are required. The current methods are mainly based on subjective measurements and therefore have several drawbacks (e.g. non continuous, intrusive) Physiological signals combined with Machine Learning (ML) methods seem an interesting approach to go beyond these limits. In this paper, we present a large scale experimentation (103 participants) where physiological data (cardiac and electrodermal activities) and subjective data (perceived VR sickness) were gathered during 30 minute VR video game sessions. Using ML methods, models were trained to predict VR sickness level (based on the physiological data labeled with the subjective data) Results showed an explained variance up to 75% (in a regression approach) and an accuracy up to 91% (in a classification approach) Despite generalization issues, this method seems promising and valuable for a real time, automatic and continuous evaluation of VR sickness, based on physiological signals and ML models.
2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
Nicolas Martin Nicolas Mathieu Nico Pallamin Martin Ragot Jean-Marc Diverrez
218597283
VR2ML: A Universal Recording and Machine Learning System for Improving Virtual Reality Experiences
We developed software for recording and reusing user actions and associated effects in various VR applications. In particular, we implemented two plugin systems that allow 3D motion data recorded in VR to be transferred to a machine learning module without programming. One is a system that runs on Unity and records actions and events in VR, and it is called "VRec." The other is a system for using data recorded with VRec for machine learning, called "VR2ML."
2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)
Yuto Yokoyama Katashi Nagao
208089660
Combining Virtual Reality and Machine Learning for Enhancing the Resiliency of Transportation Infrastructure in Extreme Events
Traffic management models that include routing choice form the basis of traffic management systems. High fidelity models that are based on rapidly evolving contextual conditions can have a significant impact on smart and energy efficient transportation. Currently, existing traffic/route choice models are generic and are calibrated on static contextual conditions. These models do not consider dynamic contextual conditions such as the location, failure of certain portions of the road network, the social network structure of population inhabiting the region (socio cultural and economic background) route choices made by other drivers, events, extreme conditions, etc. As a result, the model's predictions are made at an aggregate level and for a fixed set of contextual factors. There is a clear need to develop traffic models that take into account local contexts and are closer to ground reality to provide government agencies with the ability to make well informed model based decisions and policies.
Supratik Mukhopadhyay Yimin Zhu Ravindra Gudishala
184482902
Synesthesizer: Physical Modelling and Machine Learning for a Color Based Synthesizer in Virtual Reality
The Synesthesizer is a software synthesizer inspired to chromesthesia, that kind of synesthesia that connects sounds and colors. While chromesthesia usually produces color perception in response to sound stimulation, this synthesizer does the opposite: sound is generated according to color detection. More precisely, RGB (Red Green Blue) values are detected (one pixel at a time) and used to determine the behaviour of five physical models for virtual instruments. The motivation for creating such a synthesizer arose from the will to generate a timbral continuum out of the color continuum, allowing to explore the relation between color spectrum and sound spectrum. The Synesthesizer has two additional possible applications: A picture can become a sort of score; graphic scores can have a different source of interpretation; Given its intuitiveness, it might allow even non experts to explore the possibilities of sound synthesis.
MCM
Giovanni Santini
237299042
An Immersive Context Teaching Method for College English Based on Artificial Intelligence and Machine Learning in Virtual Reality Technology
With the development of language research and language teaching, people realize that grammatical competence is an important part of communicative competence. In foreign language teaching, grammar teaching is not only necessary but also the main way to achieve the goal of communicative competence. This article mainly studies the virtual reality technology college English immersive context teaching method based on artificial intelligence and machine learning. The purpose is to improve students' English learning ability. Through the comparative teaching experiment of two classes of freshmen in a university, the experimental class conducted VR technology based immersive virtual context teaching from the perspective of constructivism, while the control class adopted common multimedia equipment and traditional teaching methods. In the classroom, teachers occupy most of the time, students only passively receive a lot of information from teachers, they have little chance to participate in the exchange of information and express ideas in the target language, and most of the time they are "immersed" in the Chinese environment. The overall English level was also better than that of the control class, with an average score of 2.8 points higher. This shows that college English immersive context teaching combining constructivism theory and VR technology can indeed improve students' English level.
Mob. Inf. Syst.
58952279
Application Identification for Virtual Reality Video with Feature Analysis and Machine Learning Technique
Immersive media services such as Virtual Reality (VR) video have attracted more and more attention in recent years. They are applications that typically require large bandwidth, low latency, and low packet loss ratio. With limited network resources in wireless network, video application identification is crucial for optimized network resource allocation, Quality of Service (QoS) assurance, and security management. In this paper, we propose a set of statistical features that can be used to distinguish VR video from ordinary video. Six supervised machine learning (ML) algorithms are explored to verify the identification performance for VR video application using these features. Experimental results indicate that the proposed features combined with C4.5 Decision Tree algorithm can achieve an accuracy of 98.6% for VR video application identification. In addition, considering the requirement of real time traffic identification, we further make two improvements to the statistical features and training set. One is the feature selection algorithm to improve the computational performance, and the other is the study of the overall accuracy in respect to training set size to obtain the minimum training set size.
ChinaCom
Xiaoyu Liu Xinyu Chen Yumei Wang Yu Liu
220253532
Understanding research methodologies when combining virtual reality technology with machine learning techniques
Virtual Reality (VR) technology represents a new medium to provide immersive solutions in different fields. The analysis of a user while interacting in VR, through data science and machine learning (ML) techniques, might provide insights to deliver customized functionalities that enhance productivity and efficiency in learning tasks in education or rehabilitation processes in healthcare. However, empirical research involving VR often borrows methods from human computer interaction intending to evaluate human behavior through technology, whereas ML intend to create mathematical models, usually with non empirical approach. Their opposite nature might cause confusion for early stage researchers wanting to understand and follow the methodological approaches and communicative practices in empirical studies that merge both VR and ML. This paper presents a scoping review of methodological strategies undertaken in 21 peer reviewed research articles that involve both VR and ML. Results show and appraise different methodological approaches in research projects, and outline a set of recommendations to combine metrics from inferential statistics and evaluation of ML models to increase validity, reliability and trustworthiness in future research projects that intersect VR and ML.
PETRA
Luis Quintero
53778276
Machine learning distinguishes neurosurgical skill levels in a virtual reality tumor resection task
This study outlines the first investigation of application of machine learning to distinguish "skilled" and "novice" psychomotor performance during a virtual reality (VR) brain tumor resection task. Tumor resection task participants included 23 neurosurgeons and senior neurosurgery residents as the "skilled" group and 92 junior neurosurgery residents and medical students as the "novice" group. The task involved removing a series of virtual brain tumors without causing injury to surrounding tissue. Originally, 150 features were extracted followed by statistical and forward feature selection. The selected features were provided to 4 classifiers, namely, K Nearest Neighbors, Parzen Window, Support Vector Machine, and Fuzzy K Nearest Neighbors. Sets of 5 to 30 selected features were provided to the classifiers. A working point of 15 premium features resulted in accuracy values as high as 90% using the Supprt Vector Machine. The obtained results highlight the potentials of machine learning, applied to VR simulation data, to help realign the traditional apprenticeship educational paradigm to a more objective model, based on proven performance standards. Graphical abstract Using several scenarios of virtual reality neurosurgical tumor resection together with machine learning classifiers to distinguish skill level
Medical Biological Engineering Computing
Samaneh Siyar Hamed Azarnoush Saeid Rashidi Alexandre Winkler-Schwartz Vincent Bissonnette Nirros Ponnudurai Rolando F Del Maestro
229701972
Towards the Design of Adaptive Virtual Reality Horror Games: A Model of Players' Fears Using Machine Learning and Player Modeling
Horror games are designed to induce fear in players. Although fundamental fears, such as the unknown, are inherent to the human being, more specific fears, such as darkness and apparitions, are individual and can vary from person to person. When a game aims at intensifying the fear evoked in individual players, having useful information about the fears of the current player is vital to promote more frightening experiences. This paper explores fear modeling and presents a new method to identify what players fear in a virtual reality horror game. The proposed method uses machine learning and player modeling techniques to create a model of players' fears, which can be used to adapt in game horror elements to intensify the fear evoked in players. The paper presents the proposed method and evaluates its accuracy and real time performance in a virtual reality horror game.
2020 19th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames)
Edirlei Soares de Lima Bruno M C Silva Gabriel Teixeira Galam