# Lecture 18

## MGFs to get moments of Expo and Normal, sums of Poissons, joint distributions

## MGF for $Expo(1)$

Let $X \sim Expo(1)$.

We begin by finding the MGF of $X$:

\begin{align}
  M(t) &= \mathbb{E}(e^{tX}) &\quad \text{ definition of MGF} \\
       &= \int_{0}^{\infty} e^{-x} \, e^{tx} dx \\
       &= \int_{0}^{\infty} e^{-x(1-t)} dx \\
       &= \boxed{\frac{1}{1-t}} &\quad \text{ for } t < 1
\end{align}

In finding the moments, by definition we have:

* $M'(0) = \mathbb{E}(X)$
* $M''(0) = \mathbb{E}(X^2)$
* $M'''(0) = \mathbb{E}(X^3)$
* ... and so on ...

Even though finding derivatives of $\frac{1}{1-t}$ is not all that bad, it is nevertheless annoying busywork. But since we know that the $n^{th}$ moment is the coefficient of the $n^{th}$ term of the Taylor Expansion of $X$, we can leverage that fact instead.

\begin{align}
  \frac{1}{1-t} &= \sum_{n=0}^{\infty} t^n &\quad \text{ for } |t| < 1 \\ 
                &= \sum_{n=0}^{\infty} \frac{n! \, t^n}{n!} &\quad \text{ since we need the form } \sum_{n=0}^{\infty} \left( \frac{\mathbb{E}(X^{n}) \, t^{n}}{n!}\right) \\
   \\
   \Rightarrow \mathbb{E}(X^n) &= \boxed{n!}
\end{align}
 
And now we can simply _generate_ arbitary moments for r.v. $X$!

* $\mathbb{E}(X) = 1! = 1$
* $\mathbb{E}(X^2) = 2! = 2$
* $\Rightarrow \mathbb{Var}(X) = \mathbb{E}(X^2) - \mathbb{E}X^2 = 2 - 1 = 1$

## MGF for $Expo(\lambda)$
 
Let $Y \sim Expo(\lambda)$.

We begin with

\begin{align}
  \text{let } X &= \lambda Y \\
  \text{and so } X &= \lambda Y \sim Expo(1) \\
  \\
  \text{then } Y &= \frac{X}{\lambda} \\
  Y^n &= \frac{X^n}{\lambda^n} \\
  \\
  \Rightarrow \mathbb{E}(Y^n) &= \frac{\mathbb{E}(X^n)}{\lambda^n} \\
  &= \boxed{\frac{n!}{\lambda^n}}
\end{align}

And as before, we now can simply _generate_ arbitary moments for r.v. $Y$!

* $\mathbb{E}(Y) = \frac{1!}{\lambda^1} = \frac{1}{\lambda}$
* $\mathbb{E}(Y^2) = \frac{2!}{\lambda^2} = \frac{2}{\lambda^2}$
* $\Rightarrow \mathbb{Var}(Y) = \mathbb{E}(Y^2) - \mathbb{E}Y^2 = \frac{2}{\lambda^2} - \left(\frac{1}{\lambda}\right)^2 = \frac{1}{\lambda^2}$

## MGF for standard Normal $\mathcal{N}(0, 1)$

Let $Z \sim \mathcal{N}(0,1)$; find **all** its moments.

We have seen before that, by symmetry, $\mathbb{E}(Z^{2n+1}) = 0$ for all odd values of $n$.

So we will focus in on the _even_ values of $n$.

Now the MGF $M(t) = e^{t^2/2}$. Without taking _any_ derivatives, we can immediately Taylor expand that, since it is continuous everywhere.

\begin{align}
  M(t) &= e^{t^2/2} \\
       &= \sum_{n=0}^{\infty} \frac{\left(t^2/2\right)^n}{n!} \\
       &= \sum_{n=0}^{\infty} \frac{t^{2n}}{2^n \, n!} \\
       &= \sum_{n=0}^{\infty} \frac{(2n)! \, t^{2n}}{2^n \, n! \, (2n)!} &\quad \text{ since we need the form } \sum_{n=0}^{\infty} \left( \frac{\mathbb{E}(X^{n}) \, t^{n}}{n!}\right) \\
       \\
       \Rightarrow \mathbb{E}(Z^{2n}) &= \boxed{\frac{(2n)!}{2^n \, n!}}
\end{align}

Let's double-check this with what know about $\mathbb{Var}(Z)$

* by symmetry, we know that $\mathbb{E}(Z) = 0$
* at $n = 1$, $\mathbb{E}(Z^2) = \frac{2!}{2 \times 1!} = 1$ 
* $\Rightarrow \mathbb{Var}(Z) = \mathbb{E}(Z^2) - \mathbb{E}Z^2 = 1 - 0 = 1$
* at $n = 2$, $\mathbb{E}(Z^4) = \frac{4!}{4 \times 2!} = 3$
* at $n = 3$, $\mathbb{E}(Z^6) = \frac{6!}{8 \times 3!} = 15$

And so you might have noticed a pattern here. Let us rewrite those even moments once more:

* at $n = 1$, $\mathbb{E}(Z^2) = 1$ 
* at $n = 2$, $\mathbb{E}(Z^4) = 1 \times 3 = 3$ 
* at $n = 3$, $\mathbb{E}(Z^6) = 1 \times 3 \times 5 = 15$

## MGF for $Pois(\lambda)$

Let $X \sim Pois(\lambda)$; now let's consider MGFs and how to use them to find sums of random variables (convolutions).

\begin{align}
  M(t) &= \mathbb{E}(e^{tX}) \\
       &= \sum_{k=0}^{\infty} e^{tk} \, \frac{\lambda^k e^{-\lambda}}{k!} \\
       &= e^{-\lambda} \sum_{k=0}^{\infty} \frac{\lambda^k e^{tk}}{k!} \\
       &= e^{-\lambda} e^{\lambda e^t} &\quad \text{but the right is just another Taylor expansion} \\
       &= \boxed{e^{\lambda (e^t - 1)}}
\end{align}


Now let's let $Y \sim Pois(\mu)$, and it is independent of $X$. Find the distribution of $(X + Y)$.

You may recall that with MGFs, all we need to do is _multiply_ the MGFs.

\begin{align}
  e^{\lambda (e^t - 1)} \, e^{\mu (e^t - 1)} &= e^{(\lambda + \mu)(e^t - 1)} \\
  \\
  \Rightarrow X + Y &\sim \mathcal{Pois}(\lambda + \mu)
\end{align}

When adding a Poisson distribution $X$ with another independent Poisson distribution $Y$, the resulting convolution $X + Y$ will also be Poisson. This interesting relation only happens with Poisson distributions.

Now think about what happens with $X$ and $Y$ are _not_ independent.

Let $Y = X$, so that $X + Y = 2X$, which is clearly *not* Poisson, as
* $X + Y = 2X$ is an even function which cannot be Poisson, since Poisson can take on all values both even _and_ odd
* Mean $\mathbb{E}(2x) = 2\lambda$, but $\mathbb{Var}(2X) = 4\lambda$, and since the mean and variance are _not_ equal, this cannot be Poisson

### Some definitions

In the most basic case of two r.v. in a joint distribution, consider both r.v.'s _together_:

> **Joint CDF**
>
> In the general case, the joint CDF fof two r.v.'s is $ F(x,y) = P(X \le x, Y \le y)$

<p/> 

> **Joint PDF**
> $f(x, y)$ such that, in the *continuous* case $P((X,Y) \in B) = \iint_B f(x,y)\,dx\,dy$

#### Joint PMF

$f(x, y)$ such that, in the *discrete* case

\begin{align}
  P(X=x, Y=y)
\end{align}

We also can consider a single r.v. of a joint distribution:

#### Independence and Joint Distributions

$X, Y$ are independent iff $F(x,y) = F_X(x) \, F_Y(y)$. 

\begin{align}
  P(X=x, Y=y) &= P(X=x) \, P(Y=y) &\quad \text{discrete case} \\
  \\\\
  f(x, y) &= f_X(x) \, f_Y(y)     &\quad \text{continuous case}
\end{align}

... with the caveat that this must be so for *all* $\text{x, y} \in \mathbb{R}$

#### Marginals (and how to get them)

$P(X \le x)$ is the *marginal distribution* of $X$, where we consider one r.v. at a time.

In the case of a two-r.v. joint distribution, we can get the marginals by using the joint distribution itself:

\begin{align}
  P(X=x) &= \sum_y P(X=x, Y=y) &\quad \text{marginal PMF, discrete case, for } x \\
  \\\\
  f_Y(y) &= \int_{-\infty}^{\infty} f_{(X,Y)}(x,y) \, dx &\quad \text{marginal PDF, continuous case, for } y
\end{align}

## Example: Discrete Joint Distribution

Let $X, Y$ be both Bernoulli. $X$ and $Y$ may be independent; or they might be dependent. They may or may not have the same $p$. But they are both related in the form of a *joint distribution*.

We can lay out this joint distribution in a $2 \times 2$ contigency table like below:


|       | $Y=0$ | $Y=1$ |
|-------|:-----:|:-----:|
| $X=0$ |  2/6  |  1/6  |
| $X=1$ |  2/6  |  1/6  |

In order to be a joint distribution, all of the values in our contigency table must be positive; and they must all sum up to 1. The example above shows such a PMF.

Let's add the marginals for $X$ and $Y$ to our $2 \times 2$ contigency table:


|       | $Y=0$ | $Y=1$ |  ...  |
|:-----:|:-----:|:-----:|:-----:|
| $X=0$ |  2/6  |  1/6  |  3/6  |
| $X=1$ |  2/6  |  1/6  |  3/6  |
|  ...  |  4/6  |  2/6  |       |


Observe how in our example, we have:

\begin{align}
  P(X=0,Y=0) &= P(X=0) \, P(Y=0) \\
             &= 3/6 \times 4/6 = 12/36 &= \boxed{2/6} \\
  \\
  P(X=0,Y=1) &= P(X=0) \, P(Y=1) \\
             &= 3/6 \times 2/6 = 6/36 &= \boxed{1/6} \\
  P(X=1,Y=0) &= P(X=1) \, P(Y=0) \\
             &= 3/6 \times 4/6 = 12/36 &= \boxed{2/6} \\
  \\
  P(X=1,Y=1) &= P(X=1) \, P(Y=1) \\
             &= 3/6 \times 2/6 = 6/36 &= \boxed{1/6} \\
\end{align}

and so you can see that $X$ and $Y$ are independent.

Now here's an example of a two r.v. joint distribution where $X$ and $Y$ are _dependent_; check it out for yourself.

|       | $Y=0$ | $Y=1$ |
|:-----:|:-----:|:-----:|
| $X=0$ |  1/3  |   0   |
| $X=1$ |  1/3  |  1/3  |

## Example: Continuous Joint Distribution

Now say we had Uniform distributions on a square such that $x,y \in [0,1]$. 

The joint PDF would be constant on/within the square; and 0 outside.

\begin{align}
  \text{joint PDF} &=
    \begin{cases}
      c &\quad \text{if } 0 \le x \le 1 \text{, }  0 \le y \le 1 \\
      \\
      0 &\quad \text{otherwise}
    \end{cases}
\end{align}



In 1-dimension space, if you integrate $1$ over some interval you get the _length_ of that interval.

In 2-dimension space, if you integrate $1$ over some region, you get the _area_ of that region.

Normalizing $c$, we know that $c = \frac{1}{area} = 1$.

Marginally, $X$ and $Y$ are independent $\mathcal{Unif}(1)$.

## Example: Dependent, Continuous Joint Distribution

Now say we had Uniform distributions on a _disc_ such that $x^2 + y^2 \le 1$. 


#### Joint PDF

In this case, the joint PDF is $Unif$ over the area of a disc centered at the origin with radius 1.

\begin{align}
  \text{joint PDF} &=
    \begin{cases}
      \frac{1}{\pi} &\quad \text{if } x^2 + y^2 \le 1 \\
      \\
      0 &\quad \text{otherwise}
    \end{cases}
\end{align}

#### Marginal PDF



59e713edd912f7c66d601b46b3b579c1a4a57e30
ipynb
Jupyter Notebook
Lecture_18.ipynb
dirtScrapper/Stats-110-master
a123692d039193a048ff92f5a7389e97e479eb7e
BSD-3-Clause
Lecture_18.ipynb
dirtScrapper/Stats-110-master
a123692d039193a048ff92f5a7389e97e479eb7e
BSD-3-Clause
Lecture_18.ipynb
dirtScrapper/Stats-110-master
a123692d039193a048ff92f5a7389e97e479eb7e
BSD-3-Clause
Qwen/Qwen-72B
1. YES
2. YES