reinforcement learning slam keyframe
211075807
On Reward Shaping for Mobile Robot Navigation: A Reinforcement Learning and SLAM Based Approach
We present a map less path planning algorithm based on Deep Reinforcement Learning (DRL) for mobile robots navigating in unknown environment that only relies on 40 dimensional raw laser data and odometry information. The planner is trained using a reward function shaped based on the online knowledge of the map of the training environment, obtained using grid based Rao Blackwellized particle filter, in an attempt to enhance the obstacle awareness of the agent. The agent is trained in a complex simulated environment and evaluated in two unseen ones. We show that the policy trained using the introduced reward function not only outperforms standard reward functions in terms of convergence speed, by a reduction of 36.9\ of the iteration steps, and reduction of the collision samples, but it also drastically improves the behaviour of the agent in unseen environments, respectively by 23\ in a simpler workspace and by 45\ in a more clustered one. Furthermore, the policy trained in the simulation environment can be directly and successfully transferred to the real robot. A video of our experiments can be found at: this https URL
ArXiv
Nicolo Botteghi Beril Sirmacek Khaled A A Mustafa Mannes Poel Stefano Stramigioli
213630342
Path planning for active SLAM based on deep reinforcement learning under unknown environments
Autonomous navigation in complex environment is an important requirement for the design of a robot. Active SLAM (simultaneous localization and mapping) combining, which combine path planning with SLAM, is proposed to improve the ability of autonomous navigation in complex environment. In this paper, fully convolutional residual networks are used to recognize the obstacles to get depth image. The avoidance obstacle path is planned by Dueling DQN algorithm in the robot's navigation; at the same time, the 2D map of the environment is built based on FastSLAM. The experiments show that the proposed algorithm can successfully identify and avoid moving and static obstacles with different quantities in the environment, and realize the autonomous navigation of the robot in a complex environment.
Intell. Serv. Robotics
Shuhuan Wen Yanfang Zhao Xiao Yuan Zongtao Wang Dan Zhang Luigi Manfredi
221702101
REINFORCEMENT LEARNING HELPS SLAM: LEARNING TO BUILD MAPS
Abstract. In this research, we investigate the use of Reinforcement Learning (RL) for an effective and robust solution for exploring unknown and indoor environments and reconstructing their maps. We benefit from a Simultaneous Localization and Mapping (SLAM) algorithm for real time robot localization and mapping. Three different reward functions are compared and tested in different environments with growing complexity. The performances of the three different RL based path planners are assessed not only on the training environments, but also on an a priori unseen environment to test the generalization properties of the policies. The results indicate that RL based planners trained to maximize the coverage of the map are able to consistently explore and construct the maps of different indoor environments.
Nicolo Botteghi Beril Sirmacek R K Schulte Mannes Poel Christoph Brune
204956409
Unsupervised Collaborative Learning of Keyframe Detection and Visual Odometry Towards Monocular Deep SLAM
In this paper we tackle the joint learning problem of keyframe detection and visual odometry towards monocular visual SLAM systems. As an important task in visual SLAM, keyframe selection helps efficient camera relocalization and effective augmentation of visual odometry. To benefit from it, we first present a deep network design for the keyframe selection, which is able to reliably detect keyframes and localize new frames, then an end to end unsupervised deep framework further proposed for simultaneously learning the keyframe selection and the visual odometry tasks. As far as we know, it is the first work to jointly optimize these two complementary tasks in a single deep framework. To make the two tasks facilitate each other in the learning, a collaborative optimization loss based on both geometric and visual metrics is proposed. Extensive experiments on publicly available datasets \ie~KITTI raw dataset and its odometry split) clearly demonstrate the effectiveness of the proposed approach, and new state of the art results are established on the unsupervised depth and pose estimation from monocular videos.
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Lu Sheng Dan Xu Wanli Ouyang Xiaogang Wang
16974115
Reinforcement Learning SLAM for finding minimum cost path and mapping
In this work, we propose the integration of two of the most widely used approaches for the implementation of autonomous navigation systems: the reinforcement learning for path finding, along with SLAM (Simultaneous Localization and Mapping) type algorithms for localization and mapping of the environment. These two approaches are integrated to address the problem of how a robot should explore an unknown and dynamic environment while it collects perception features in order to locate itself and, at the same time, to obtain information clues about cost traversability of an area. So, when a robot is exploring and mapping with a SLAM algorithm it is also learning to associate perception features with costs and actions to find optimal paths from the starting point to the goal point in dynamical environments.
World Automation Congress 2012
Nancy Arana-Daniel Roberto Rosales-Ochoa Carlos Lopez-Franco Emmanuel Nuno
229406034
A Deep Reinforcement Learning Approach for Active SLAM
In this paper, we formulate the active SLAM paradigm in terms of model free Deep Reinforcement Learning, embedding the traditional utility functions based on the Theory of Optimal Experimental Design in rewards, and therefore relaxing the intensive computations of classical approaches. We validate such formulation in a complex simulation environment, using a state of the art deep Q learning architecture with laser measurements as network inputs. Trained agents become capable not only to learn a policy to navigate and explore in the absence of an environment model but also to transfer their knowledge to previously unseen maps, which is a key requirement in robotic exploration.
Julio A Placed Jose A Castellanos
195434979
TOWARDS CONTINUOUS CONTROL FOR MOBILE ROBOT NAVIGATION: A REINFORCEMENT LEARNING AND SLAM BASED APPROACH
Abstract. We introduce a new autonomous path planning algorithm for mobile robots for reaching target locations in an unknown environment where the robot relies on its on board sensors. In particular, we describe the design and evaluation of a deep reinforcement learning motion planner with continuous linear and angular velocities to navigate to a desired target location based on deep deterministic policy gradient (DDPG) Additionally, the algorithm is enhanced by making use of the available knowledge of the environment provided by a grid based SLAM with Rao Blackwellized particle filter algorithm in order to shape the reward function in an attempt to improve the convergence rate, escape local optima and reduce the number of collisions with the obstacles. A comparison is made between a reward function shaped based on the map provided by the SLAM algorithm and a reward function when no knowledge of the map is available. Results show that the required learning time has been decreased in terms of number of episodes required to converge, which is 560 episodes compared to 1450 episodes in the standard RL algorithm, after adopting the proposed approach and the number of obstacle collision is reduced as well with a success ratio of 83% compared to 56% in the standard RL algorithm. The results are validated in a simulated experiment on a skid steering mobile robot.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences
Khaled A A Mustafa Nicolo Botteghi Beril Sirmacek Mannes Poel Stefano Stramigioli
211057820
Real time RGB D semantic keyframe SLAM based on image segmentation learning from industrial CAD models
This paper presents methods for performing realtime semantic SLAM aimed at autonomous navigation and control of a humanoid robot in a manufacturing scenario. A novel multi keyframe approach is proposed that simultaneously minimizes a semantic cost based on class level features in addition to common photometric and geometric costs. The approach is shown to robustly construct a 3D map with associated class labels relevant to robotic tasks. Alternatively to existing approaches, the segmentation of these semantic classes have been learnt using RGB D sensor data aligned with an industrial CAD manufacturing model to obtain noisy pixel wise labels. This dataset confronts the proposed approach in a complicated real world setting and provides insight into the practical use case scenarios. The semantic segmentation network was fine tuned for the given use case and was trained in a semi supervised manner using noisy labels. The developed software is real time and integrated with ROS to obtain a complete semantic reconstruction for the control and navigation of the HRP4 robot. Experiments in situ at the Airbus manufacturing site in Saint Nazaire validate the proposed approach.
2019 19th International Conference on Advanced Robotics (ICAR)
Howard Mahe Denis Marraud Andrew I Comport
145992290
Reinforcement Learning and SLAM based approach for mobile robot navigation in unknown environments
Autonomous navigation of robots in unknown environments from their current position to a desired target without colliding with obstacles represents an important aspect in the field of mobile robots. In literature, traditional methods do exist in case a complete knowledge of the environment is available. However, this is not the case in real life applications where a complete knowledge about stochastic environments can hardly be obtained where the positions of the obstacles are unknown. Our main goal is to navigate a skid steering mobile robot (SSMR) with non holonomic constraints in an unknown environment to its desired target in minimum time while avoiding colliding with the obstacles. In the context of autonomous cart navigation, autonomous navigation of mobile robots in unstructured environments can be formulated as a Reinforcement learning (RL) problem. Reinforcement learning is a paradigm in which the agent (robot) learns its optimal path by interacting with the environment and receiving a reward based on its actions. Based on this action reward scenario, the optimal action for each state can be discovered by maximizing a predefined accumulated reward that reflects the quality of the trajectory taken by the robot. Reinforcement learning can be categorized into two main methods (Bagnell J. Peters J. Kober, 2013) namely value based methods (Q learning, Deep Q learning) and policy based methods (REINFORCE with Policy Gradient) (i) Value based methods: In these methods, the value function that maps each state action pair into a value is learned. Thanks to these methods, the best action to take for each state, the action with the biggest value, can be found. This works well in case of a finite set of actions. (ii) Policy based methods: In policy based methods, instead of learning value functions that give an indication of the expected total reward for each state action pair, the policy function that maps the state into action is optimized directly (Brundage. M Arulkumaran. K and A. 2017) Policy search methods are more effective in high dimensional action spaces, or when using continuous actions like the case of mobile robot navigation (Neumann G. Peters J. Deisenroth, 2013) However, since both of these methods have their own drawbacks, there is hybrid method called Actor Critic that employs both value functions and policy search (Brundage. M Arulkumaran. K and A. 2017) In this method, the actor adjust the policy parameters by a policy gradient ascent whereas the critic estimates the action value function using a policy evaluation algorithm such as temporal difference (TD) learning. This can be done through two neural networks running in parallel. Herein, we introduce a reinforcement learning based navigation system that enhance the navigation capabilities of a skid steering mobile robot. This goal will be achieved by using asynchronous deep deterministic policy gradient (ADDPG) through an actor critic algorithm (Badia A. Mirza M. Graves A. Harley T. Lillicrap T. Silver D. Kavukcuoglu K. Mnih, 2016) Two deep neural networks are introduced in this case. The first one (actornetwork) transfers the input vector that represents the states of the robot to linear and angular velocity commands that represent the actions. Whereas the second neural network (critic network) is responsible for predicting the Q value of the state and action pairs. Deterministic policies will be utilized since it was proposed that they outperform their stochastic counterparts in highdimensional action spaces (Lever G. Hess N. Degris T. Wierstra D. Riedmiller M. Silver, 2014) On the other hand, Simultaneous Localization and Mapping also known as SLAM techniques are to be integrated with reinforcement learning in an attempt to improve the learning rate by providing more accurate estimation of the robots states. SLAM has been an active topic for many years (C. Cadena, 2016) because it provides two fundamental components to robotics: where I am, and what I see. Herein, we introduce how the navigation problem of non holonomic mobile robots can be formulated as a reinforcement learning problem that could be solved by using ADDPG actor critic algorithm. Besides, we share the experimental results on how SLAM would help reinforcement learning at all by comparing the learning rate without SLAM (comparison when a partial map of the environment is given and when no map is available) For implementation of the parallel running neural networks, we do our initial experiments using the GPU cluster of University of Twente. However, when it comes to real life implementations and demonstrations on real robots, the hardware becomes the key component to determine whether the goal can be achieved or not. Therefore, we implement the trained networks on NVIDIA Jetson TX2 in order to show real life implementation possibilities when the algorithms need to run in a navigating robot (Jetson TX2 Module, online documentation, n.d. In this sense, NVIDIA Jetson TX2 gives opportunity to run even multiple complex neural networks real time on a credic card size processor unit which can easily be carried by a small (less than 20cm length) robot. We show our experiments on sensor selection and the OpenAI package in ROSGazebo platform as a possible simulation environment. Last but not least, we discuss the real life implementation scenarios using our NVIDIA Jetson TX2 and how realistic the simulation results could be when the algorithms are implemented on the real robot. The early experiments indicate that, for an optimal design of a SLAM and reinforcement learning based system to truly work in real world applications in order to reach a goal in an unknown environment, the hardware setup and the algorithms must be collaboratively designed.
Beril Sirmacek Nicolo Botteghi Mustafa I A Khaled Stefano Stramigioli Mannes Poel
9508914
Learning to Prevent Monocular SLAM Failure using Reinforcement Learning
Monocular SLAM refers to using a single camera to estimate robot ego motion while building a map of the environment. While Monocular SLAM is a well studied problem, automating Monocular SLAM by integrating it with trajectory planning frameworks is particularly challenging. This paper presents a novel formulation based on Reinforcement Learning (RL) that generates fail safe trajectories wherein the SLAM generated outputs do not deviate largely from their true values. Quintessentially, the RL framework successfully learns the otherwise complex relation between perceptual inputs and motor actions and uses this knowledge to generate trajectories that do not cause failure of SLAM. We show systematically in simulations how the quality of the SLAM dramatically improves when trajectories are computed using RL. Our method scales effectively across Monocular SLAM frameworks in both simulation and in real world experiments with a mobile robot.
ICVGIP
Vignesh Prasad Saurabh Singh Nahas Pareekutty Balaraman Ravindran K Madhava Krishna