Introduction to Probability

Definition 1

A probability space is a triple $(\Omega, \mathcal{F}, P)$ where $\Omega$ is a set of objects called the sample space, $\mathcal{F}$ is a family of subsets of $\Omega$ called events, and the probability measure $P:\mathcal{F}\rightarrow [0,1]$.

One key assumption we make is that $\mathcal{F}$ is a $\sigma$-algebra containing $\Omega$, meaning that countably many complements, unions, and intersections of events in $\mathcal{F}$ are also events in $\mathcal{F}$. The probability measure $P$ must obey Kolmogorov’s Axioms.

1. $\forall A \in \mathcal{F},\ P(A) \geq 0$

2. $P(\Omega) = 1$

3. If $A_1, A_2, \cdots\in \mathcal{F}$ and $\forall i\ne j,\ A_i\bigcap A_j=\emptyset$, then $P\left(\bigcup_{i\geq 1}A_i\right) = \sum_{i\geq1}P(A_i)$

We choose $\Omega$ and $\mathcal{F}$ to model problems in a way that makes our calculations easy.

Theorem 1

$P(A^c) = 1 - P(A)$

Theorem 2 (Inclusion-Exclusion Principle)

$P\left( \bigcup_{i=1}^{n}A_i \right) = \sum_{k=1}^{n}(-1)^{k+1}\left( \sum_{1\leq i_1<\cdots<i_k\leq n} P(A_{i_1}\cap \cdots \cap A_{i_k}) \right)$

Theorem 3 (Law of Total Probability)

If $A_1, A_2, \cdots$ partition $\Omega$ (i.e $A_i$ are disjoint and $\cup A_i = \Omega$), then for event $B$,

$P(B) = \sum_iP(B\cap A_i)$

Conditional Probability

Definition 2

If $B$ is an event with $P(B)>0$, then the conditional probability of $A$ given $B$ is

$P(A|B) = \frac{P(A\cap B)}{P(B)}$

Intuitively, conditional probabilty is the probability of event $A$ given that event $B$ has occurred. In terms of probability spaces, it is as if we have taken $(\Omega, \mathcal{F}, P)$ and now have a probabilty measure $P(\cdot|C)$ belonging to the space $(\Omega, \mathcal{F}, P(\cdot|C))$.

Theorem 4 (Bayes Theorem)

$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$

Independence

Definition 3

Events $A$ and $B$ are independent if $P(A\cap B) = P(A)P(B)$

If $P(B)>0$, then $A, B$ are independent if and only if $P(A|B) = P(A)$. In other words, knowing $B$ occurred gave no extra information about $A$.

Definition 4

If $A,B,C$ with $P(C)>0$ satisfy $P(A\cap B|C) = P(A|C)P(B|C)$, then $A$ and $B$ are conditionally independent given $C$.

Conditional independence is a special case of independence where $A$ and $B$ are not necessarily independent in the original probability space which has the measure $P$, but are independent in the new probability space conditioned on $C$ with the measure $P(\cdot|C)$.