Probability is associated with at least one event. E.g.: rolling a die or pulling a coloured ball out of a bag

Outcome of the event is random (can’t be sure of the outcome of the die will). 

Hence the variable that represents the outcome of these events is called a random variable

# A. 3 Types of Probability

## Marginal Probability

Marginal Probability is the probability of event A occurring, P(A). Rigorously written as P(X=A).

Example: A pack of playing cards, the probability that a card drawn from a pack is red: P(X=red) = 0.5

## Joint Probability

Joint Probability is the probability ofntersection of two or more events. Written as $ P(A \cap B) , P(A,B), P(A\ and\ B)$

Example: Probability that a card drawn from a pack is red and 4 is P(red and 4) = 2/52 = 1/26

<b>NOTE: Joint probability is symmetrical P(A,B) = P(B,A)</b>

## Conditional Probability

Conditional probability is the probability of an event(s) occuring given that we know other events have already occurred. If A and B are two events then the conditional probability of A occurring given that B has occurred is written as P(A|B)

Example: Probability that a card is a four given that we have drawn a red card is P(4|red) = 2/26 = 1/13

# B. Linking the 3 types of Probability

This is the equation linking the 3 probabilities
\begin{equation}
\begin{aligned}
P(A | B) =  \dfrac{P(A \cap B)}{P(B)}
\end{aligned}
\tag{Equation B.1}
\end{equation}

### Intutition

We start with. This is logical.
\begin{equation}
\begin{aligned}
P(A \cap B) =  P(B)\,P(A \mid B)
\end{aligned}
\tag{Equation B.2}
\end{equation}

Rearranging eq B.2, we get the following

\begin{equation}
\begin{aligned}
P(A \cap B) =  P(A \mid B)\,P(B)
\end{aligned}
\tag{Equation B.3}
\end{equation}
\begin{align}
\end{align}

Rearranging further we get the first equation B.1

# C. Special Case of Joint Probabilities for Independent Events

When two events A and B are independent, P(A | B) = P(A)
Using this in equation B.3 we get

\begin{equation}
\begin{aligned}
P(A \cap B) =  P(A)\,P(B)
\end{aligned}
\tag{Equation C.1}
\end{equation}
\begin{align}
\end{align}

# D. Additive rule of Probability

Addition is used in OR scenario we have to add the individual probabilities and subtract the intersection

\begin{equation}
\begin{aligned}
P(A \cup B) =  P(A) + P(B) - P(A \cap B)
\end{aligned}
\tag{Equation D.1}
\end{equation}
\begin{align}
\end{align}

If you visualize Venn Diagram, then adding P(A) and P(B) accounts for adding the intersection twice. One of this has to be removed. Hence the equation

## D.1 Special case of Mutually exclusive events
In case of mutually exclusive events $ P(A \cap B) = 0 $ and the above equation simplifies as 

\begin{equation}
\begin{aligned}
P(A \cup B) =  P(A) + P(B)
\end{aligned}
\tag{Equation D.2}
\end{equation}
\begin{align}
\end{align}


# E. Bayes Theorem

Bayes theorem provides a mechanism to utilize prior knowledge to improve probability estimation

E.g: 
\begin{align*}
\text{Initially: P(Person has Cancer) = Percent of population with cancer}
\end{align*}

\begin{align*}
\text{Prior evidence: Person is smoker}
\end{align*}

\begin{align*}
\text{Modification: } P(\text{Person has Cancer}\, \mid \text{Person is smoker})\, > \,  P(\text{Person has Cancer)}
\end{align*}


\begin{equation}
\begin{aligned}
P(A \mid B) =  \dfrac{P(B \mid A)\,P(A)}{P(B)}
\end{aligned}
\tag{Equation E.1}
\end{equation}

It is also used when calculating one conditional propbability say P(A|B) is hard, but the converse conditional probability P(B|A) is easily found

E.g.:
1. Suppose there are 2 bowls - M and N. 
2. Bowl M has 5 Oranges and 8 Apples.  
3. Bowl N has 12 Oranges and 4 Apples

What is probability of having picked from M, given than the picking was Apple ?
\begin{align*}
\text{Modification: } P(M \mid Apple)\, = \, \dfrac{P(Apple \mid N)\,P(M)}{P(Apple)}
\end{align*}

Here, the P(LHS) is hard by itself. But all P() on RHS can be easily calculated. Hence its utlity

## E.1 Derivation

P(A,B) is equal to P(B,A). Combining b.2 and B.3 
\begin{equation}
P(B)\,P(A \mid B) =  P(A \mid B)\,P(B)
\end{equation}

Re arranging, we get Bayes Theorem forumla shown in E.1

## E.2 Bayesian Terminology

1. P(A|B) is called the posterior; this is what we are trying to estimate. In the above example, this would be the “probability of having cancer given that the person is a smoker”.
2. P(B|A) is called the likelihood; this is the probability of observing the new evidence, given our initial hypothesis. In the above example, this would be the “probability of being a smoker given that the person has cancer”.
3. P(A) is called the prior; this is the probability of our hypothesis without any additional prior information. In the above example, this would be the “probability of having cancer”.
4. P(B) is called the marginal likelihood; this is the total probability of observing the evidence. In the above example, this would be the “probability of being a smoker”. In many applications of Bayes Rule, this is ignored, as it mainly serves as normalization.

\begin{align*}
P(A \mid B)\, \propto \,P(B \mid A)\,P(A)
\end{align*}

# Acknowledgements
1. Probability Basics Explanation https://towardsdatascience.com/probability-concepts-explained-introduction-a7c0316de465
2. https://machinelearningmastery.com/joint-marginal-and-conditional-probability-for-machine-learning/
3. https://towardsdatascience.com/what-is-bayes-rule-bb6598d8a2fd
4. https://machinelearningmastery.com/bayes-theorem-for-machine-learning/
5. https://towardsdatascience.com/maximum-likelihood-vs-bayesian-estimation-dd2eb4dfda8a



```python

```

49ff1b642b8b68e1042b890672b0defc3b813e09
ipynb
Jupyter Notebook
Bayesian/01-Probability_Intro.ipynb
datavector-io/datascience
b1de0cd1c563b3c90d3f4382f0130c77e18308f9
MIT
Bayesian/01-Probability_Intro.ipynb
datavector-io/datascience
b1de0cd1c563b3c90d3f4382f0130c77e18308f9
MIT
Bayesian/01-Probability_Intro.ipynb
datavector-io/datascience
b1de0cd1c563b3c90d3f4382f0130c77e18308f9
MIT
Qwen/Qwen-72B
1. YES
2. YES