[ML UTD 24] Machine Learning Up-To-Date — Life With Data
The purpose of this article is to clearly explain Q-Learning from the perspective of a Bayesian. As such, we use a small grid world and a simple extension of tabular Q-Learning to illustrate the fundamentals. Specifically, we show how to extend the deterministic Q-Learning algorithm to model the variance of Q-values with Bayes’ rule. We focus on a sub-class of problems where it is reasonable to assume that Q-values are normally distributed and derive insights when this assumption holds true. Lastly, we demonstrate that applying Bayes’ rule to update Q-values comes with a challenge: it is vulnerable to early exploitation of suboptimal policies.

This article is largely based on the seminal work from Dearden et al. Specifically, we expand on the assumption that Q-values are normally distributed and evaluate various Bayesian exploration policies. One key distinction is that we model μ and σ 2, while the authors of the original Bayesian Q-Learning paper model a distribution over these parameters. This allows them to quantify uncertainty in their parameters as well as the expected return — we only focus on the latter.

… keep reading
['Anthony Agnone']
['Artificial Intelligence', 'Technology', 'Machine Learning', 'Data Science', 'Programming']