# GDA

```{note}
Gaussian Discriminant Analysis(GDA) suppose that observations are presumed to come from one of several multivariate normal distributions.<br/>
It is 
```

## Normal Distribution

Normal distribution is parameterized by a mean vector $\mu \in \mathbb{R}^{d}$ and a covariance matrix $\Sigma \in \mathbb{R}^{d \times d}$, where $\Sigma >= 0$ is symmetric and positive semi-definite, also written $\mathcal{N}(\mu,\Sigma)$, it's density is given by：

$$p(x;\mu,\Sigma)=\frac{1}{(2\pi)^{d/2}\left | \Sigma \right |^{1/2} }\exp\left ( -\frac{1}{2}(x-\mu)^{T}{\Sigma}^{-1}(x-\mu)\right )$$

unsurprisingly, for random variable $X \sim \mathcal{N}(\mu,\Sigma)$:

$$E[X] = \int_{x}xp(x;\mu, \Sigma)dx = \mu$$
$$Cov(X) = E[(X - E(X))(X - E(X))^{T}] = \Sigma$$

## Model

When we have a classification problem in which the input features $x$ are continous, we can use GDA, which model $p(x|y)$ using the multivariant normal distribution:

$$y\sim Bernoulli(\phi)$$
$$x | y=0 \sim \mathcal{N}(\mu_{0},\Sigma) $$
$$x | y=1 \sim \mathcal{N}(\mu_{1},\Sigma) $$

density:

$$p(y) = \phi^{y}(1 - \phi)^{1 - y}$$

$$p(x| y=0)=\frac{1}{(2\pi)^{d/2}\left | \Sigma \right |^{1/2} }\exp\left (-\frac{1}{2}(x-\mu_{0})^{T}{\Sigma}^{-1}(x-\mu_{0})\right )$$

$$p(x| y=1)=\frac{1}{(2\pi)^{d/2}\left | \Sigma \right |^{1/2} }\exp\left (-\frac{1}{2}(x-\mu_{1})^{T}{\Sigma}^{-1}(x-\mu_{1})\right )$$

Then the log-likelihood of the data is given by：

$$
\begin{equation}
\begin{split}
l(\phi,\mu_{0},\mu_{1},\Sigma) &= \log\prod_{i=1}^{n}p(x^{(i)},y^{(i)};\phi,\mu_{0},\mu_{1},\Sigma) \\
&= \log\prod_{i=1}^{n}p(x^{(i)}|y^{(i)};\mu_{0},\mu_{1},\Sigma)p(y^{(i)};\phi)
\end{split}
\end{equation}
$$

Maximum Likelihood Estimate result：

$$\phi = \frac{1}{n}\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}$$

$$\mu_{0} = \frac{\sum_{i=1}^{n}1\left \{ y^{(i)}=0 \right \}x^{(i)}  }{\sum_{i=1}^{n}1\left \{ y^{(i)}=0 \right \}} $$

$$\mu_{1} = \frac{\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}x^{(i)}  }{\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}} $$

$$\Sigma=\frac{1}{n}\sum_{i=1}^{n}(x^{(i)} - \mu_{y^{(i)}})(x^{(i)} - \mu_{y^{(i)}})^{T}$$


```python

```

b1b37218d334f0e4598cc7dba54a9c4098e1acd6
ipynb
Jupyter Notebook
machine-learning-book/c3.GDA.ipynb
newfacade/jupyters
12d3c8bf1b91a7fc2f84e89b5a55efa176f4da23
MIT
machine-learning-book/c3.GDA.ipynb
newfacade/jupyters
12d3c8bf1b91a7fc2f84e89b5a55efa176f4da23
MIT
machine-learning-book/c3.GDA.ipynb
newfacade/jupyters
12d3c8bf1b91a7fc2f84e89b5a55efa176f4da23
MIT
Qwen/Qwen-72B
1. YES
2. YES