Introduction to Probability

Definition 1

A probability space is a triple
(Ω,F,P)(\Omega, \mathcal{F}, P)
where
Ω\Omega
is a set of objects called the sample space,
F\mathcal{F}
is a family of subsets of
Ω\Omega
called events, and the probability measure
P:F[0,1]P:\mathcal{F}\rightarrow [0,1]
.
One key assumption we make is that
F\mathcal{F}
is a
σ\sigma
-algebra containing
Ω\Omega
, meaning that countably many complements, unions, and intersections of events in
F\mathcal{F}
are also events in
F\mathcal{F}
. The probability measure
PP
must obey Kolmogorov’s Axioms.
  1. 1.
    AF, P(A)0\forall A \in \mathcal{F},\ P(A) \geq 0
  2. 2.
    P(Ω)=1P(\Omega) = 1
  3. 3.
    If
    A1,A2,FA_1, A_2, \cdots\in \mathcal{F}
    and
    ij, AiAj=\forall i\ne j,\ A_i\bigcap A_j=\emptyset
    , then
    P(i1Ai)=i1P(Ai)P\left(\bigcup_{i\geq 1}A_i\right) = \sum_{i\geq1}P(A_i)
We choose
Ω\Omega
and
F\mathcal{F}
to model problems in a way that makes our calculations easy.

Theorem 1

P(Ac)=1P(A)P(A^c) = 1 - P(A)

Theorem 2 (Inclusion-Exclusion Principle)

P(i=1nAi)=k=1n(1)k+1(1i1<<iknP(Ai1Aik))P\left( \bigcup_{i=1}^{n}A_i \right) = \sum_{k=1}^{n}(-1)^{k+1}\left( \sum_{1\leq i_1<\cdots<i_k\leq n} P(A_{i_1}\cap \cdots \cap A_{i_k}) \right)

Theorem 3 (Law of Total Probability)

If
A1,A2,A_1, A_2, \cdots
partition
Ω\Omega
(i.e
AiA_i
are disjoint and
Ai=Ω\cup A_i = \Omega
), then for event
BB
,
P(B)=iP(BAi)P(B) = \sum_iP(B\cap A_i)

Conditional Probability

Definition 2

If
BB
is an event with
P(B)>0P(B)>0
, then the conditional probability of
AA
given
BB
is
P(AB)=P(AB)P(B)P(A|B) = \frac{P(A\cap B)}{P(B)}
Intuitively, conditional probabilty is the probability of event
AA
given that event
BB
has occurred. In terms of probability spaces, it is as if we have taken
(Ω,F,P)(\Omega, \mathcal{F}, P)
and now have a probabilty measure
P(C)P(\cdot|C)
belonging to the space
(Ω,F,P(C))(\Omega, \mathcal{F}, P(\cdot|C))
.

Theorem 4 (Bayes Theorem)

P(AB)=P(BA)P(A)P(B)P(A|B) = \frac{P(B|A)P(A)}{P(B)}

Independence

Definition 3

Events
AA
and
BB
are independent if
P(AB)=P(A)P(B)P(A\cap B) = P(A)P(B)
If
P(B)>0P(B)>0
, then
A,BA, B
are independent if and only if
P(AB)=P(A)P(A|B) = P(A)
. In other words, knowing
BB
occurred gave no extra information about
AA
.

Definition 4

If
A,B,CA,B,C
with
P(C)>0P(C)>0
satisfy
P(ABC)=P(AC)P(BC)P(A\cap B|C) = P(A|C)P(B|C)
, then
AA
and
BB
are conditionally independent given
CC
.
Conditional independence is a special case of independence where
AA
and
BB
are not necessarily independent in the original probability space which has the measure
PP
, but are independent in the new probability space conditioned on
CC
with the measure
P(C)P(\cdot|C)
.