Berkeley Notes
  • Introduction
  • EE120
    • Introduction to Signals and Systems
    • The Fourier Series
    • The Fourier Transform
    • Generalized transforms
    • Linear Time-Invariant Systems
    • Feedback Control
    • Sampling
    • Appendix
  • EE123
    • The DFT
    • Spectral Analysis
    • Sampling
    • Filtering
  • EECS126
    • Introduction to Probability
    • Random Variables and their Distributions
    • Concentration
    • Information Theory
    • Random Processes
    • Random Graphs
    • Statistical Inference
    • Estimation
  • EECS127
    • Linear Algebra
    • Fundamentals of Optimization
    • Linear Algebraic Optimization
    • Convex Optimization
    • Duality
  • EE128
    • Introduction to Control
    • Modeling Systems
    • System Performance
    • Design Tools
    • Cascade Compensation
    • State-Space Control
    • Digital Control Systems
    • Cayley-Hamilton
  • EECS225A
    • Hilbert Space Theory
    • Linear Estimation
    • Discrete Time Random Processes
    • Filtering
  • EE222
    • Real Analysis
    • Differential Geometry
    • Nonlinear System Dynamics
    • Stability of Nonlinear Systems
    • Nonlinear Feedback Control
Powered by GitBook
On this page
  • Conditional Probability
  • Independence

Was this helpful?

  1. EECS126

Introduction to Probability

PreviousEECS126NextRandom Variables and their Distributions

Last updated 3 years ago

Was this helpful?

Definition 1

A probability space is a triple (Ω,F,P)(\Omega, \mathcal{F}, P)(Ω,F,P) where Ω\OmegaΩ is a set of objects called the sample space, F\mathcal{F}F is a family of subsets of Ω\OmegaΩ called events, and the probability measure P:F→[0,1]P:\mathcal{F}\rightarrow [0,1]P:F→[0,1].

One key assumption we make is that F\mathcal{F}F is a σ\sigmaσ-algebra containing Ω\OmegaΩ, meaning that countably many complements, unions, and intersections of events in F\mathcal{F}F are also events in F\mathcal{F}F. The probability measure PPP must obey Kolmogorov’s Axioms.

  1. ∀A∈F, P(A)≥0\forall A \in \mathcal{F},\ P(A) \geq 0∀A∈F, P(A)≥0

  2. P(Ω)=1P(\Omega) = 1P(Ω)=1

  3. If A1,A2,⋯∈FA_1, A_2, \cdots\in \mathcal{F}A1​,A2​,⋯∈F and ∀i≠j, Ai⋂Aj=∅\forall i\ne j,\ A_i\bigcap A_j=\emptyset∀i=j, Ai​⋂Aj​=∅, then P(⋃i≥1Ai)=∑i≥1P(Ai)P\left(\bigcup_{i\geq 1}A_i\right) = \sum_{i\geq1}P(A_i)P(⋃i≥1​Ai​)=∑i≥1​P(Ai​)

We choose Ω\OmegaΩ and F\mathcal{F}F to model problems in a way that makes our calculations easy.

Theorem 1

P(Ac)=1−P(A)P(A^c) = 1 - P(A)P(Ac)=1−P(A)

Theorem 2 (Inclusion-Exclusion Principle)

P(⋃i=1nAi)=∑k=1n(−1)k+1(∑1≤i1<⋯<ik≤nP(Ai1∩⋯∩Aik))P\left( \bigcup_{i=1}^{n}A_i \right) = \sum_{k=1}^{n}(-1)^{k+1}\left( \sum_{1\leq i_1<\cdots<i_k\leq n} P(A_{i_1}\cap \cdots \cap A_{i_k}) \right)P(⋃i=1n​Ai​)=∑k=1n​(−1)k+1(∑1≤i1​<⋯<ik​≤n​P(Ai1​​∩⋯∩Aik​​))

Theorem 3 (Law of Total Probability)

If A1,A2,⋯A_1, A_2, \cdotsA1​,A2​,⋯ partition Ω\OmegaΩ (i.e AiA_iAi​ are disjoint and ∪Ai=Ω\cup A_i = \Omega∪Ai​=Ω), then for event BBB,

P(B)=∑iP(B∩Ai)P(B) = \sum_iP(B\cap A_i)P(B)=∑i​P(B∩Ai​)

Conditional Probability

Definition 2

Theorem 4 (Bayes Theorem)

Independence

Definition 3

Definition 4

If BBB is an event with P(B)>0P(B)>0P(B)>0, then the conditional probability of AAA given BBB is

P(A∣B)=P(A∩B)P(B)P(A|B) = \frac{P(A\cap B)}{P(B)}P(A∣B)=P(B)P(A∩B)​

Intuitively, conditional probabilty is the probability of event AAA given that event BBB has occurred. In terms of probability spaces, it is as if we have taken (Ω,F,P)(\Omega, \mathcal{F}, P)(Ω,F,P) and now have a probabilty measure P(⋅∣C)P(\cdot|C)P(⋅∣C) belonging to the space (Ω,F,P(⋅∣C))(\Omega, \mathcal{F}, P(\cdot|C))(Ω,F,P(⋅∣C)).

P(A∣B)=P(B∣A)P(A)P(B)P(A|B) = \frac{P(B|A)P(A)}{P(B)}P(A∣B)=P(B)P(B∣A)P(A)​

Events AAA and BBB are independent if P(A∩B)=P(A)P(B)P(A\cap B) = P(A)P(B)P(A∩B)=P(A)P(B)

If P(B)>0P(B)>0P(B)>0, then A,BA, BA,B are independent if and only if P(A∣B)=P(A)P(A|B) = P(A)P(A∣B)=P(A). In other words, knowing BBB occurred gave no extra information about AAA.

If A,B,CA,B,CA,B,C with P(C)>0P(C)>0P(C)>0 satisfy P(A∩B∣C)=P(A∣C)P(B∣C)P(A\cap B|C) = P(A|C)P(B|C)P(A∩B∣C)=P(A∣C)P(B∣C), then AAA and BBB are conditionally independent given CCC.

Conditional independence is a special case of independence where AAA and BBB are not necessarily independent in the original probability space which has the measure PPP, but are independent in the new probability space conditioned on CCC with the measure P(⋅∣C)P(\cdot|C)P(⋅∣C).