Many real-world control applications, from economics to robotics, are characterized by the presence of multiple conicting objectives. In these problems, the standard concept of optimality is replaced by Pareto{optimality and the goal is to nd the Pareto frontier, a set of solutions representing dierent compromises among the objectives. Despite recent advances in multi{objective optimization, achieving an accurate representation of the Pareto frontier is still an important challenge. In this paper, we propose a reinforcement learning policy gradient approach to learn a continuous approximation of the Pareto frontier in multi{objective Markov Decision Problems (MOMDPs). Dierently from previous policy gradient algorithms, where n optimization routines are executed to have n solutions, our approach performs a single gradient ascent run, generating at each step an improved continuous approximation of the Pareto frontier. The idea is to optimize the parameters of a function dening a manifold in the policy parameters space, so that the corresponding image in the objectives space gets as close as possible to the true Pareto frontier. Besides deriving how to compute and estimate such gradient, we will also discuss the non{trivial issue of dening a metric to assess the quality of the candidate Pareto frontiers. Finally, the properties of the proposed approach are empirically evaluated on two problems, a linear-quadratic Gaussian regulator and a water reservoir control task.
['Simone Parisi', 'Matteo Pirotta', 'Marcello Restelli']
Multi-objective Reinforcement Learning through Continuous Pareto Manifold Approximation