reinforcement learning for ball
204873020
Transferring Deep Reinforcement Learning from a Game Engine Simulation for Robots
Simulations are used to gather data for machine learning algorithms at low cost. However, many robot simulators can not render realistic graphics. Realistic image data is essential for several robots using reinforcement learning algorithms in fields such as self driving cars, agricultural weed detection and grasping objects. In this report, we propose the use of modern game engines with highly realistic rendering for simulating robots and training them with deep reinforcement learning using image data. We successfully simulated and trained a Turtlebot2 robot in Unity3D with Deep Q learning to find and drive into a blue ball. The resulting reinforcement learning model was used to control a real Turtlebot2 robot. The simulated and real robot are interchangeable by design, making it easy to use the reinforcement algorithm to control either. The real robot was controllable by the Q learning algorithm, but not able to perform the task. The use of modern game engines for simulation of robots for reinforcement learning is shown to be promising. However, for future work, testing of more realistic simulation environments are needed to assess the usability of game engines for realistically simulating robots.
Christoffer B Lillelund
53109335
Reinforcement Learning of Phase Oscillators for Fast Adaptation to Moving Targets
Online movement generation in tasks involving real humanoid robots interacting with fast moving targets is extremely difficult. This paper approaches this problem via imitation and reinforcement learning using phase variables. Imitation learning is used to acquire primitive trajectories of the demonstrator interacting with the target. The temporal progress of the robot is represented as a function of the target's phase. Using a phase oscillator formulation, reinforcement learning optimizes a temporal policy such that the robot can quickly react to large/unexpected changes in the target movement. The phase representation decouples the temporal and spatial problems allowing the use of fast online solutions. The methodology is applicable in both cyclic and single stroke movements. We applied the proposed method on a real bi manual humanoid upper body with 14 degrees of freedom where the robot had to repeatedly push a ball hanging in front of it. In simulation, we show a human robot interaction scenario where the robot changed its role from giver to receiver as a function of the interaction reward.
CoRL
Guilherme J Maeda Okan Koc Jun Morimoto
23366943
A Reinforcement Learning Fuzzy Controller for the Ball and Plate system
In this paper, a new fuzzy logic controller, namely Reinforcement Learning Fuzzy Controller (RLFC) is proposed and implemented. Based on fuzzy logic, this newly proposed online learning controller is capable of improving its behavior by learning from experiences it gains through interaction with the plant. RLFC is well established for hardware implementation with or without a priori knowledge about the plant. To evidence this claim, a hardware implementation of Ball and Plate system was established, and RLFC was then developed and applied to it. The obtained results are illustrated in this paper.
International Conference on Fuzzy Systems
Nima Mohajerin Mohammad Bagher Menhaj Ali Doustmohammadi
13163246
A Fuzzy Reinforcement Learning for a Ball Interception Problem
In this paper, we propose a reinforcement learning method called a fuzzy Q learning where an agent determines its action based on the inference result by a fuzzy rule based system. We apply the proposed method to a soccer agent that intercepts a passed ball by another agent. In the proposed method, the state space is represented by internal information the learning agent maintains such as the relative velocity and the relative position of the ball to the learning agent. We divide the state space into several fuzzy subspaces. A fuzzy if then rule in the proposed method represents a fuzzy subspace in the state space. The consequent part of the fuzzy if then rules is a motion vector that suggests the moving direction and velocity of the learning agent. A reward is given to the learning agent if the distance between the ball and the agent becomes smaller or if the agent catches up with the ball. It is expected that the learning agent finally obtains the efficient positioning skill.
RoboCup
Tomoharu Nakashima Masayo Udo Hisao Ishibuchi
155109324
Direct Policy Search Reinforcement Learning Based on Variational Bayesian Inference
Direct policy search is a promising reinforcement learning framework in particular for controlling continuous, high dimensional systems. As one of direct policy search, reward weighted regression (RWR) was proposed by Peters et al. The RWR algorithm estimates the policy parameter based on EM algorithm and is therefore prone to overfitting. In this paper, we focus on variational Bayesian inference to avoid overfitting problem, and propose the direct policy search reinforcement learning based on variational Bayesian inference (VBRL) The performance of the proposed VBRL is assessed in several experiments with mountain car and ball batting task. These experiments highlight the VBRL produces higher average return and outperforms the RWR.
2018 Joint 10th International Conference on Soft Computing and Intelligent Systems (SCIS) and 19th International Symposium on Advanced Intelligent Systems (ISIS)
Nobuhiko Yamaguchi Kazuya Ihara Osamu Fukuda Hiroshi Okumura
232373770
A Validation Approach for Deep Reinforcement Learning of a Robotic Arm in a 3D Simulated Environment
In recent years, deep reinforcement learning has increasingly contributed to the development of robotic applications and boosted research in robotics. Deep learning and model free, off policy, value based reinforcement learning algorithms enabled agents to successfully learn complex robotic skills through trial and error process and visual inputs. The aim of this paper concerns the training of a robot in a simulation environment by designing a Deep Q Network (DQN) that elaborates images acquired by an RGB vision sensor inside a 3D simulated environment and outputs a value for each action the robotic arm can execute given the current state. In particular, the robot has to push a ball into a soccer net without any knowledge of the environment and its own location. In addition, our further goal was to perform agent validation during training and assess its generalization level. Despite the many advances in reinforcement learning, it is still a challenge. Therefore, we devised a validation strategy similar to the method applied in supervised learning and tested the agent both on known and unknown experiences, achieving interesting and promising results.
2021 IEEE 19th World Symposium on Applied Machine Intelligence and Informatics (SAMI)
Monica Gruosso Nicola Felice Capece Ugo Erra Flavio Biancospino
227209222
Real time Active Vision for a Humanoid Soccer Robot Using Deep Reinforcement Learning
In this paper, we present an active vision method using a deep reinforcement learning approach for a humanoid soccer playing robot. The proposed method adaptively optimises the viewpoint of the robot to acquire the most useful landmarks for self localisation while keeping the ball into its viewpoint. Active vision is critical for humanoid decision maker robots with a limited field of view. To deal with an active vision problem, several probabilistic entropy based approaches have previously been proposed which are highly dependent on the accuracy of the self localisation model. However, in this research, we formulate the problem as an episodic reinforcement learning problem and employ a Deep Q learning method to solve it. The proposed network only requires the raw images of the camera to move the robot's head toward the best viewpoint. The model shows a very competitive rate of 80% success rate in achieving the best viewpoint. We implemented the proposed method on a humanoid robot simulated in Webots simulator. Our evaluations and experimental results show that the proposed method outperforms the entropy based methods in the RoboCup context, in cases with high self localisation errors.
ICAART
Soheil Khatibi Meisam Teimouri Mahdi Rezaei
61153600
Simultaneously Learning Vision and Feature based Control Policies for Real world Ball in a Cup
We present a method for fast training of vision based control policies on real robots. The key idea behind our method is to perform multi task Reinforcement Learning with auxiliary tasks that differ not only in the reward to be optimized but also in the state space in which they operate. In particular, we allow auxiliary task policies to utilize task features that are available only at training time. This allows for fast learning of auxiliary policies, which subsequently generate good data for training the main, vision based control policies. This method can be seen as an extension of the Scheduled Auxiliary Control (SAC X) framework. We demonstrate the efficacy of our method by using both a simulated and real world Ball in a Cup game controlled by a robot arm. In simulation, our approach leads to significant learning speed ups when compared to standard SAC X. On the real robot we show that the task can be learned from scratch, i.e. with no transfer from simulation and no imitation learning. Videos of our learned policies running on the real robot can be found at this https URL.
Robotics: Science and Systems
Devin Schwab Jost Tobias Springenberg Murilo Fernandes Martins Thomas Lampe Michael Neunert Abbas Abdolmaleki Tim Hertweck Roland Hafner Francesco Nori Martin A Riedmiller
226970143
Ball and Beam Control using Adaptive PID based on Q Learning
The ball and beam system is one of the most used systems for benchmarking the controller response because it has nonlinear and unstable characteristics. Furthermore, in line with the increasing of computation power availability and artificial intelligence research intensity, especially the reinforcement learning field, nowadays plenty of researchers are working on a learning control approach for controlling systems. Due to that, in this paper, the adaptive PID controller based on Q Learning (Q PID) was used to control the ball position on the ball and beam system. From the simulation result, Q PID outperforms the conventional PID and heuristic PID controller technique with the swifter settling time and lower overshoot percentage.
2020 7th International Conference on Electrical Engineering, Computer Sciences and Informatics (EECSI)
Brilian Putra Amiruddin Rusdhianto Effendi Abdul Kadir
211842415
Learning a Robot to Score a Penalty Minimal Reward Reinforcement Learning
Reinforcement Learning is applied to learn a robot how to score a penalty in simulation. Minimal rewards are given to let the robot learn as much as possible by itself. Intermediate rewards are applied to reduce the amount of trials needed to learn a good action policy up to 75% without effecting the accuracy. Furthermore, different methods of combining object states (multiplying and additive) are examined to determine the method that uses less trials to learn. The additive method results in a decrease of 58% in the amount of trials. A robot is able to learn how to score a penalty against a standing goalkeeper within an average of 170 trials using all intermediate rewards, the additive state representation method and sixteen different states for both the ball and the goalkeeper, with an average accuracy of 69%
Caitlin Lagrand