# 1D Maze with Q Learning

In this notebook we attempt to solve the 1D Maze reinforcement learning problem. However, instead of using _TD Learning_ we will be using _Q Learning_. The difference is in the value function and the delta function. When using Q learning, one considers the value of a __state action pair__. Being in a state and going left has a different value than being in a state and going right. The Q function and delta function look like so: <br>
$\boldsymbol{Q}(\boldsymbol{s},\boldsymbol{a}) = \boldsymbol{r}(\boldsymbol{s} + 1) + \gamma \boldsymbol{r}(\boldsymbol{s} + 2) \ldots \gamma^{\boldsymbol{n}-1} \boldsymbol{r}(\boldsymbol{s}+\boldsymbol{n})$ <br>
$\delta(\boldsymbol{s},\boldsymbol{a}) = (\boldsymbol{r}(\boldsymbol{s}+1) + \gamma \boldsymbol{Q}(\boldsymbol{s}+1, \boldsymbol{a}+1)) - \boldsymbol{Q}(\boldsymbol{s}, \boldsymbol{a})$ <br>

The Q Table for a maze of length four would look like the following:
<!--
##### 4 State:
| State: | 1 | 2 | 3 | 4 |
| --- | --- | --- | --- | --- |
| Left | 1/2 | 1 | 1/2 | 1/4 |
| Right | 1/2 | 1/4 | 1/2 | 1 |
-->

##### 6 State:
| State: | 0 | 1 | 2 | 3 | 4 | 5 |
| --- | --- | --- | --- | --- | --- | --- |
| Left | 0 | 1 | 1/2 | 1/4 | 1/8 | 1/4 |
| Right | 0 | 1/4 | 1/8 | 1/4 | 1/2 | 1 |





```python
import numpy as np
from sympy import *
init_printing(use_latex=True)
from fractions import Fraction

EPOCH = 100
GAMMA = .5
EPSILON = .5
class Maze:
    def __init__(self, goal, length):
        self.reward = np.zeros(length)
        self.value = np.zeros((2,length))
        self.goal = goal
        self.length = length
        
        self.reward[goal] = 1
        
    # Performs one episode of learning on the maze
    # S is the starting state
    def episode(self, s):
         #get action
        if (self.Q(s,0) >= self.Q(s,1)):
            a = 0
        else:
            a = 1
        while s != self.goal:
            self.value[a, s] += self.delta(s,a)
            left, right = self.orient(s)
            if (a == 0):
                s = left
            else:
                s = right
            # Epsilon soft
            if (np.random.random() > EPSILON):
                if (self.Q(s,0) >= self.Q(s,1)):
                    a = 0
                else:
                    a = 1
            else:
                a = np.random.choice([0,1])
            
    # delta(s,a) = r(s+1) + GAMMA * maxQ(s+1,a+1) - Q(s,a)
    def delta(self, s, a):
        return  self.Q(s, a) - self.value[a, s]
    
    # Q(s,a) = r(s+1) + GAMMA * Q(s+1, a+1)
    def Q(self, s, a):
        # Get next state and action
        left, right = self.orient(s)
        if (a == 0):
            nextS = left
        else:
            nextS = right
        
        # Get value
        if (s == self.goal):
            # no future moves (only reward)
            return self.reward[nextS]
        else:
            if (self.value[0, nextS] >= self.value[1, nextS]):
                nextValue = self.value[0, nextS]
            else:
                nextValue = self.value[1, nextS]
            return self.reward[nextS] + GAMMA * nextValue
        

    def orient(self, s):
        # Obtain which state is to the left and which is to the right
        if (s == 0):
            left = self.length-1
            right = s+1
        elif (s == self.length-1):
            left = s-1
            right = 0
        else:
            left = s-1
            right = s+1
 
        return left, right
        
    
    # Print the maze 
    def display(self):
        print("L: ", end='')
        for i in self.value[0]:
            print( "| %3s" %Fraction(i), end='')
        print("|")
        print("R: ", end='')
        for i in self.value[1]:
            print( "| %3s" %Fraction(i), end='')
        print("|")
        print()
    
```


```python
maze = Maze(0, 6)
print("The new matrix is:")
maze.display()

for i in range(EPOCH):
    s = np.random.randint(0,maze.length)
    maze.episode(s)
print("The trained matrix is:")
maze.display()
```

    The new matrix is:
    L: |   0|   0|   0|   0|   0|   0|
    R: |   0|   0|   0|   0|   0|   0|
    
    The trained matrix is:
    L: |   0|   1| 1/2| 1/4| 1/8| 1/4|
    R: |   0| 1/4| 1/8| 1/4| 1/2|   1|
    



```python

```

561fca5c69b0b1620f5cdf23d57be34d88701e54
ipynb
Jupyter Notebook
Q-Learning 1D Maze.ipynb
mtsu-cs-summer-research/neural-networks
9f3dbc25736fb80b3c52a6630f4f61c1361a349d
MIT
Q-Learning 1D Maze.ipynb
mtsu-cs-summer-research/neural-networks
9f3dbc25736fb80b3c52a6630f4f61c1361a349d
MIT
Q-Learning 1D Maze.ipynb
mtsu-cs-summer-research/neural-networks
9f3dbc25736fb80b3c52a6630f4f61c1361a349d
MIT
Qwen/Qwen-72B
1. YES
2. YES