multimodal visual tactile fusion for  robot perception
106539300
Designing and developing virtual and augmented environments 6th International Conference, VAMR 2014, held as part of HCI International 2014, Heraklion, Crete, Greece, June 22 27, 2014, Proceedings. Part I
Classification of Interaction Techniques in the 3D Virtual Environment on Mobile Devices. Multimodal Interfaces and Sensory Fusion in VR for Social Interactions. Multi Modal Interaction System to Tactile Perception. Principles of Dynamic Display Aiding Presence in Mixed Reality Space Design. Combining Multi Sensory Stimuli in Virtual Worlds. A Progress Report. R V Dynamics Illusion Psychophysical Influence on Sense of Weight by Mixed Reality Visual Stimulation of Moving Objects. Expansion of the Free Form Projection Display Using a Hand Held Projector. Study of an Interactive and Total Immersive Device with a Personal 3D Viewer and its Effects on the Explicit Long term Memories of the Subjects. Simulation Research on Virtual Movement Based on Kinect. A Natural User Interface for Navigating in Organized 3D Virtual Requirements for Virtualization of AR Displays within VR Environments. Robot Behavior for Enhanced Human Performance and Workload. Subjective Situational Study of Presence. Development of a Squad Level Vocabulary for Human Robot Interaction. Towards an Interaction Concept for Efficient Control of Cyber physical Systems. 3D Design for Augmented Reality. Don't Walk into Walls: Creating and Visualizing Consensus Realities for Next Generation Videoconferencing. Transparency in a Human Machine Context: Approaches for Fostering Shared Awareness/Intent. Delegation and Transparency: Coordinating Interactions So Information Exchange is No Surprise. Trust and Consequences: A Visual Perspective. Choosing a Selection Technique for a Virtual Augmented. Virtual Humans for Interpersonal and Communication Skills.
Randall Shumaker Stephanie J Lackey
28231304
A framework for the fusion of visual and tactile modalities for improving robot perception
Robots should ideally perceive objects using human like multi modal sensing such as vision, tactile feedback, smell, and hearing. However, the features presentations are different for each modal sensor. Moreover, the extracted feature methods for each modal are not the same. Some modal features such as vision, which presents a spatial property, are static while features such as tactile feedback, which presents temporal pattern, are dynamic. It is difficult to fuse these data at the feature level for robot perception. In this study, we propose a framework for the fusion of visual and tactile modal features, which includes the extraction of features, feature vector normalization and generation based on bag of system (BoS) and coding by robust multi modal joint sparse representation (RM JSR) and classification, thereby enabling robot perception to solve the problem of diverse modal data fusion at the feature level. Finally, comparative experiments are carried out to demonstrate the performance of this framework.Chuang Xin Dian Ti Chu Liao Yi Chong Shi Hong Jue Xin Xi Rong He Kuang Jia He Lu Bang Duo Mo Tai Lian He Xi Shu Biao Shi Bian Ma Fang Fa Jie Jue You Yu Ji Qi Ren Gan Zhi De Shi (Jing Tai Hong Jue (Dong Tai )Kua Mo Tai Xin Xi Te Zheng Kong Jian Wei Du Bu Tong Er Dai Lai De Te Zheng Ceng Rong He Nan Ti Ju Ti Bao Gua :Shi Hong Jue Te Zheng Ti Qu Yong "Ci Dai "Suan Fa Gui Yi Hua Wei Du Bu Tong De Te Zheng Xiang Liang Lu Bang Duo Mo Tai Lian He Xi Shu Biao Shi Bian Ma Tong Guo Shi Hong Jue Rong He Suan Fa Jin Xing Fen Lei
Science China Information Sciences
Wenchang Zhang Fuchun Sun Hang Wu Haolin Yang
219980696
Grasp State Assessment of Deformable Objects Using Visual Tactile Fusion Perception
Humans can quickly determine the force required to grasp a deformable object to prevent its sliding or excessive deformation through vision and touch, which is still a challenging task for robots. To address this issue, we propose a novel 3D convolution based visual tactile fusion deep neural network (C3D VTFN) to evaluate the grasp state of various deformable objects in this paper. Specifically, we divide the grasp states of deformable objects into three categories of sliding, appropriate and excessive. Also, a dataset for training and testing the proposed network is built by extensive grasping and lifting experiments with different widths and forces on 16 various deformable objects with a robotic arm equipped with a wrist camera and a tactile sensor. As a result, a classification accuracy as high as 99.97% is achieved. Furthermore, some delicate grasp experiments based on the proposed network are implemented in this paper. The experimental results demonstrate that the C3D VTFN is accurate and efficient enough for grasp state assessment, which can be widely applied to automatic force control, adaptive grasping, and other visual tactile spatiotemporal sequence learning problems.
2020 IEEE International Conference on Robotics and Automation (ICRA)
Shaowei Cui Rui Wang Junhang Wei Fanrong Li Shuo Wang
220939386
Self Attention Based Visual Tactile Fusion Learning for Predicting Grasp Outcomes
Predicting whether a particular grasp will succeed is critical to performing stable grasping and manipulating tasks. Robots need to combine vision and touch as humans do to accomplish this prediction. The primary problem to be solved in this process is how to learn effective visual tactile fusion features. In this letter, we propose a novel Visual Tactile Fusion learning method based on the Self Attention mechanism (VTFSA) to address this problem. We compare the proposed method with the traditional methods on two public multimodal grasping datasets, and the experimental results show that the VTFSA model outperforms traditional methods by a margin of 5+ and 7+ Furthermore, visualization analysis indicates that the VTFSA model can further capture some position related visual tactile fusion features that are beneficial to this task and is more robust than traditional methods.
IEEE Robotics and Automation Letters
Shaowei Cui Rui Wang Junhang Wei Jingyi Hu Shuo Wang
196018922
Visual Tactile Fusion Material Identification Using Dictionary Learning
It is highly expected to develop methodologies capable of integrating information from visual and tactile modality in order to improve the performance of perception. However, real multimodal data often suffers from significant weak pairing characteristics; i.e. the full pairing between data samples may not be known, while a group of samples from one modality is paired to a group of samples in another modality is known. In this chapter, a novel projective dictionary learning framework is established for weakly paired multimodal data fusion. By introducing a latent pairing matrix, the simultaneous dictionary learning and pairing matrix estimation are realized, which improves the fusion effect. In addition, the kernelized version and the optimization algorithms are also addressed. Extensive experimental validations on some existing datasets are performed to show the advantages of the proposed method.
Huaping Liu Fuchun Sun
208979182
Robot Visual Tactile Information Fusion Drives Visual Attention
This paper describes a theoretical model of visual tactile information fusion mode to drive visual attention. According to the visual and tactile information obtained by the robot, the corresponding position of the most important object and the stimulation signal point for the current task is determined, and the visual attention is focused on the target object of interest. According to the error information fed back by the sensor, the relevant parameters are continuously corrected, and the robustness of the system is improved. The theory was tested on a robot with two degrees of freedom on the head and two degrees of freedom on the arm. The test results show that the model can adapt to changes in itself and the environment and has good robustness. Introduction Humans receive a large amount of information from the visual, tactile, auditory, and olfactory at every moment in the course of their homework or behavior. They drive the head by comparing the saliency of the obtained information so that the region of interest is located. The high resolution foveal area of the retina allows attention to the area so that accurate visual perception of the area can be achieved. This is called selective attention in cognitive physiology. Studies have shown that humans and many animals have mechanisms that can fuse multimodal information to stimulate visually selective attention and quickly retrieve regions of interest. Regions of interest are the areas that are most interesting and best represent the content of the image [1 3] The precise visual perception of the region is to obtain local features of the image, eliminate interference from secondary information, and obtain more accurate image content An intelligent robot with a multi mode sensing system can collect a large amount of sensor information, which lays the foundation for the robot to issue the correct instructions. However, processing a large amount of information reduces its response speed. How to improve the response speed of robots based on making full use of sensor information resources has become a hot topic in robot research. Imitating human multi modal information fusion to drive visual attention mechanism, using robot multi modal information fusion to drive visual attention is an effective way to solve this problem. It can use sensor resources more fully and reasonably, and take advantage of visual information to make the information obtained by visual sensors more accurate and more valuable. Avoid processing a large amount of information unrelated to the current task, and improve the efficiency of information processing. In order to adapt the robot system to the changes of the surrounding optical environment and the wear of the mechanical system, simulate the process of infant intelligence development, use the feedback sensor information, correct relevant parameters in real time, and compensate for errors caused by changes in the external and own conditions. Selective attention is one of the hot spots in the field of machine vision research. At present, there are a large number of related literatures on visual information driven visual attention [4 6] using the color, brightness, texture and other related information collected by the visual sensor to calculate the corresponding saliency and drive attention shift. According to the characteristics of the salient points, the region of interest close to the salient point features is retrieved, and the region of interest is image processed, which reduces the area of image processing and improves the efficiency of computer image information processing. In recent years, neurophysiological research has obtained some research results on multi modal information fusion to drive visual attention. In particular, the latest research results in brain science indicate that there are regions of multi modal
DEStech Transactions on Engineering and Technology Research
Jian-gang Sun Tao Yu Chun-chun Ma Hui Wang
115596080
Robotic Material Perception Using Active Multimodal Fusion
Robotic material perception is an extremely important but challenging problem for industrial intelligence. The main difficulties come from the fact that the material properties are difficult to be comprehensibly evaluated by single visual, auditory, or tactile modality. Conventional multimodal fusion methods require collecting all of the multimodal information for a testing sample before the recognition. This is expensive, redundant, and may incur large latency. To tackle this problem, a new active fusion framework for the multimodal material recognition is proposed in this paper. We first adopt the adversarial learning method to obtain the modal invariant representations to effectively bridge the gap between different modalities and then develop a reinforcement learning method for active modality selection. The developed framework and algorithms are evaluated on a publicly available dataset and show promising material recognition results. The developed framework provides an effective method for industrial material inspection.
IEEE Transactions on Industrial Electronics
Huaping Liu Fuchun Sun Xinyu Zhang
218940429
Multimodal visual image processing of mobile robot in unstructured environment based on semi supervised multimodal deep network
With the continuous development of computer technology, machine vision and image processing algorithms, people's research on mobile robots with vision systems is becoming deeper and deeper. This paper studies the related problems of visual image processing of mobile robots in outdoor unstructured environments. In this work, we propose a new approach that integrates heterogeneous features through a well designed Semi supervised multimodal deep network (SMMDN) For each modality, there is a multi layer sub neural network with a separate structure corresponding to it, which is used to transform features in different modes into the same modal features. At the same time, through a network layer common to all modes above these sub neural networks, a connection is established between these different modes, and finally a plurality of heterogeneous modes is converted into the same mode and a plurality of them are extracted from fusion characteristics of data modalities. The simulation results prove that SMMDN improves the perception and recognition ability of mobile robots for outdoor complex environments.
J. Ambient Intell. Humaniz. Comput.
Yajia Li
212549962
Knowledge Based Multimodal Intention Active Perception Algorithm Research
With the aging of the population, more and more old people need to live alone at home. In order to improve the robot's ability to understand the user's intention in the home environment, this paper proposes a knowledge based multimodal intention active perception algorithm. The robot actively perceives the user's intention to interact before the user issues interactive commands. Finally, the algorithm was verified on Pepper robot. On the basis of knowledge, this algorithm can actively understand the user's intention by integrating multimodal information. First of all, visual intention inference and auditory intention inference are obtained by combining database and expert knowledge base. Then, the fusion results are obtained by using the multimodal intention active perception algorithm. The robot initiatively initiates interactions to request user feedback and perform corresponding actions. Experimental results show that the proposed multimode algorithm is superior to the single mode algorithm. It improves the efficiency of human computer interaction and reduces the burden of users, and is well received by test users.
ICCDE
Jie Yuan Zhiquan Feng Jinglan Tian Xue Fan Qingbei Guo
72940880
Learning to Identify Object Instances by Touch: Tactile Recognition via Multimodal Matching
Much of the literature on robotic perception focuses on the visual modality. Vision provides a global observation of a scene, making it broadly useful. However, in the domain of robotic manipulation, vision alone can sometimes prove inadequate: in the presence of occlusions or poor lighting, visual object identification might be difficult. The sense of touch can provide robots with an alternative mechanism for recognizing objects. In this paper, we study the problem of touch based instance recognition. We propose a novel framing of the problem as multi modal recognition: the goal of our system is to recognize, given a visual and tactile observation, whether or not these observations correspond to the same object. To our knowledge, our work is the first to address this type of multi modal instance recognition problem on such a large scale with our analysis spanning 98 different objects. We employ a robot equipped with two GelSight touch sensors, one on each finger, and a self supervised, autonomous data collection procedure to collect a dataset of tactile observations and images. Our experimental results show that it is possible to accurately recognize object instances by touch alone, including instances of novel objects that were never seen during training. Our learned model outperforms other methods on this complex task, including that of human volunteers.
2019 International Conference on Robotics and Automation (ICRA)
Justin Lin Roberto Calandra Sergey Levine