EECS 126 Full Course Notes

Lecture 1: Axioms of Probability

EECS 126 (UC Berkeley) · Spring 2026 · January 20, 2026

A probability space provides the mathematical framework for modeling random experiments. In this lecture we define the axioms of probability, introduce sample spaces and events, work through several foundational examples, and derive the birthday paradox bound.

Probability Spaces

Definition 1.1 (Probability space)

A probability space is a triple $(\Omega, \FF, \PP)$ where:

$\Omega$ is the sample space: the set of all possible outcomes of the experiment.
$\FF \subseteq 2^{\Omega}$ is a $\sigma$ -algebra of subsets of $\Omega$ , called events.
$\PP : \FF \to [0,1]$ is a probability measure satisfying the Kolmogorov axioms.

The elements $\omega \in \Omega$ are called base outcomes (or samples). A $\sigma$ -algebra $\FF$ is a family of subsets of $\Omega$ that is closed under countable unions, countable intersections, and complements, and that contains $\Omega$ itself. In many discrete settings we simply take $\FF = 2^{\Omega}$ (the power set), so that every subset of $\Omega$ is an event.

Example 1.1 (Tossing two fair coins)

Toss two fair coins. The sample space is

\Omega = \{HH,\; HT,\; TH,\; TT\}.

Each base outcome is equally likely, so we assign $\PP(\{\omega\}) = \tfrac{1}{4}$ for each $\omega \in \Omega$ . The event space is the power set $\FF = 2^{\Omega}$ , which has $|\FF| = 2^{|\Omega|} = 2^4 = 16$ elements.

Example 1.2 (Toss until heads)

Toss a coin repeatedly until the first heads. Then

\Omega = \{H,\; TH,\; TTH,\; TTTH,\; \ldots\}

is a countably infinite sample space.

Example 1.3 (Waiting for a bus)

Wait at a bus stop for the next bus, which arrives at some time in $[0, T]$ . Now

\Omega = [0, T]

is an uncountably infinite (continuous) sample space. In this setting the choice of $\FF$ becomes nontrivial; one typically takes $\FF$ to be the Borel $\sigma$ -algebra on $[0,T]$ .

Kolmogorov Axioms

Definition 1.2 (Kolmogorov axioms)

A function $\PP : \FF \to \RR$ is a probability measure on $(\Omega, \FF)$ if it satisfies:

Nonnegativity. $\PP(A) \ge 0$ for all $A \in \FF$ .
Normalization. $\PP(\Omega) = 1$ .
Countable additivity. If $A_1, A_2, \ldots \in \FF$ are pairwise disjoint ( $A_i \cap A_j = \varnothing$ for $i \ne j$ ), then $\PP\!\Bigl(\bigcup_{i \ge 1} A_i\Bigr) = \sum_{i \ge 1} \PP(A_i).$

Why countable additivity? In a continuous sample space such as $\Omega = [0,1]$ , every individual point has probability zero: $\PP(\{x\}) = 0$ . If we only had finite additivity, we could not recover $\PP([0,1]) = 1$ from the probabilities of individual points. Countable additivity bridges this gap and is essential for the theory of continuous distributions.

Example 1.4 (Biased coin)

Flip a biased coin: heads with probability $p$ , tails with probability $1 - p$ . Set

\Omega = \{H, T\}, \qquad \FF = 2^{\Omega} = \bigl\{\varnothing,\; \{H\},\; \{T\},\; \{H,T\}\bigr\}.

Define

\PP(H) = p, \quad \PP(T) = 1-p, \quad \PP(\varnothing) = 0, \quad \PP(\{H,T\}) = 1.

Then $(\Omega, \FF, \PP)$ is a valid probability space.

Fundamental Probability Facts

The following basic identities are direct consequences of the Kolmogorov axioms.

Theorem 1.1 (Complement rule)

For any event $A \in \FF$ ,

\PP(A^c) = 1 - \PP(A).

Proof.

Since $A$ and $A^c$ are disjoint with $A \cup A^c = \Omega$ , countable additivity gives $\PP(A) + \PP(A^c) = \PP(\Omega) = 1$ .

Theorem 1.2 (Inclusion–exclusion)

For any two events $A, B \in \FF$ ,

\PP(A \cup B) = \PP(A) + \PP(B) - \PP(A \cap B).

More generally, for events $A_1, A_2, \ldots, A_n$ ,

\begin{aligned} \PP\!\Bigl(\bigcup_{i=1}^{n} A_i\Bigr) = \sum_{i} \PP(A_i) &- \sum_{i < j} \PP(A_i \cap A_j) + \sum_{i < j < k} \PP(A_i \cap A_j \cap A_k) \\ &- \cdots + (-1)^{n+1}\, \PP(A_1 \cap \cdots \cap A_n). \end{aligned}

Theorem 1.3 (Union bound)

For any events $A_1, \ldots, A_n$ ,

\PP\!\Bigl(\bigcup_{i=1}^{n} A_i\Bigr) \;\le\; \sum_{i=1}^{n} \PP(A_i).

Proof.

Define $B_1 = A_1$ and $B_k = A_k \setminus \bigcup_{i<k} A_i$ for $k \ge 2$ . The $B_k$ are disjoint with $\bigcup B_k = \bigcup A_k$ , and $\PP(B_k) \le \PP(A_k)$ , so $\PP(\bigcup A_k) = \sum \PP(B_k) \le \sum \PP(A_k)$ .

Discrete Probability and Counting

In a discrete sample space, the probability of an event is just the sum of the probabilities of its base outcomes:

\PP(A) = \sum_{\omega \in A} \PP(\{\omega\}).

Definition 1.3 (Uniform sample space)

When all outcomes are equally likely, we say $\Omega$ is a uniform sample space, and for any event $A$ ,

\PP(A) = \frac{|A|}{|\Omega|}.

The Birthday Paradox

Example 1.5 (Birthday paradox)

Let $n$ people be in a room. What is the probability that at least two share a birthday? Let $K = 365$ and assume birthdays are uniform. Let $A$ = “at least two people share a birthday.” Compute via the complement:

\PP(A^c) = \frac{K(K-1)(K-2)\cdots(K-n+1)}{K^n} = \prod_{j=0}^{n-1}\Bigl(1 - \frac{j}{K}\Bigr).

Using $1 - x \le e^{-x}$ :

\PP(A^c) \le \prod_{j=0}^{n-1} e^{-j/K} = e^{-\sum_{j=0}^{n-1} j/K} = e^{-n(n-1)/(2K)}.

Theorem 1.4 (Birthday bound)

$\PP(A) \ge 1 - e^{-n(n-1)/(2K)}$ . For $K = 365$ and $n = 23$ , this gives $\PP(A) \ge 50\%$ .

Balls-and-bins interpretation. The birthday problem is equivalent to throwing $n$ balls uniformly into $K$ bins and asking for the probability of a collision. This connection arises frequently in hashing, load balancing, and networking.

Lecture 2: Conditional Probability and Independence

EECS 126 (UC Berkeley) · Spring 2026 · January 22, 2026

In this lecture we introduce conditional probability, derive the multiplication (chain) rule, state the law of total probability and Bayes’ rule, work through the classic false-positive medical testing example, and then develop the theory of independence and conditional independence.

Conditional Probability

Definition 2.1 (Conditional probability)

Let $A, B$ be events with $\PP(B) > 0$ . The conditional probability of $A$ given $B$ is

\PP(A \mid B) = \frac{\PP(A \cap B)}{\PP(B)}.

Intuitively, $\PP(A \mid B)$ is the probability that $A$ occurs when we know that $B$ has occurred. Conditioning on $B$ restricts the sample space from $\Omega$ to $B$ ; the event $B$ becomes the new “ $\Omega$ ” for $A$ .

In a uniform sample space, $\PP(A) = |A|/|\Omega|$ . Conditioning on $B$ gives $\PP(A \mid B) = |A \cap B|/|B|$ , which is simply the fraction of outcomes in $B$ that also lie in $A$ .

Example 2.1 (Two dice – conditioning on the sum)

Roll two standard dice. Let $A = \{\text{1st die is 6}\}$ , $B = \{\text{sum of two dice is 7}\}$ , and $C = \{\text{sum of two dice is 11}\}$ .

Computing $\PP(A \mid B)$ : The outcomes in $B$ (sum = 7) are $\{(1,6),(2,5),(3,4),(4,3),(5,2),(6,1)\}$ , so $\PP(B) = 6/36$ . The only outcome in $A \cap B$ is $(6,1)$ , so $\PP(A \cap B) = 1/36$ . Therefore

\PP(A \mid B) = \frac{1/36}{6/36} = \frac{1}{6}.

Computing $\PP(A \mid C)$ : The outcomes in $C$ (sum = 11) are $\{(5,6),(6,5)\}$ , so $\PP(C) = 2/36$ . The only outcome in $A \cap C$ is $(6,5)$ , so

\PP(A \mid C) = \frac{1/36}{2/36} = \frac{1}{2}.

Multiplication Rule (Chain Rule)

Theorem 2.1 (Chain rule of probability)

For any events $A_1, A_2, \ldots, A_n$ with $\PP(A_1 \cap A_2 \cap \cdots \cap A_{n-1}) > 0$ ,

\PP(A_1 \cap A_2 \cap \cdots \cap A_n) = \PP(A_1)\,\PP(A_2 \mid A_1)\,\PP(A_3 \mid A_1 \cap A_2) \cdots \PP(A_n \mid A_1 \cap \cdots \cap A_{n-1}).

Proof.

By repeated application of the definition of conditional probability. Write $\PP(A_1 \cap A_2) = \PP(A_1)\,\PP(A_2 \mid A_1)$ , then $\PP(A_1 \cap A_2 \cap A_3) = \PP(A_1 \cap A_2)\,\PP(A_3 \mid A_1 \cap A_2)$ , and continue inductively. Each factor is well-defined because $\PP(A_1 \cap \cdots \cap A_{k-1}) > 0$ implies $\PP(A_1 \cap \cdots \cap A_k) \le \PP(A_1 \cap \cdots \cap A_{k-1})$ , and we are given the intersection has positive probability up to $n-1$ terms.

Birthday paradox via the chain rule. The chain rule gives an alternative derivation of the birthday bound. Let $A_i = \{\text{person } i \text{ doesn't share a birthday with persons } 1, \ldots, i-1\}$ . Then

\PP(A_1 \cap \cdots \cap A_n) = 1 \cdot \frac{K-1}{K} \cdot \frac{K-2}{K} \cdots \frac{K-(n-1)}{K} = \prod_{j=0}^{n-1}\Bigl(1 - \frac{j}{K}\Bigr) \approx e^{-n(n-1)/(2K)},

recovering the same bound from Lecture 1.

Law of Total Probability

Theorem 2.2 (Law of total probability)

Let $A_1, A_2, \ldots, A_N$ be a partition of $\Omega$ (i.e. the $A_i$ are pairwise disjoint — mutually exclusive — and their union is $\Omega$ — collectively exhaustive). Then for any event $B$ ,

\PP(B) = \sum_{i=1}^{N} \PP(A_i)\,\PP(B \mid A_i).

Proof.

Since $\{A_i\}$ partitions $\Omega$ , we can decompose $B$ as $B = \bigcup_{i=1}^N (A_i \cap B)$ , and these pieces are disjoint. By countable additivity, $\PP(B) = \sum_{i=1}^N \PP(A_i \cap B) = \sum_{i=1}^N \PP(A_i)\,\PP(B \mid A_i)$ , where the last step uses the definition of conditional probability.

Bayes’ Rule

Theorem 2.3 (Bayes’ rule)

Let $A_1, \ldots, A_N$ partition $\Omega$ . Given an event $B$ with $\PP(B) > 0$ ,

\PP(A_i \mid B) = \frac{\PP(B \mid A_i)\,\PP(A_i)}{\sum_{j=1}^{N} \PP(B \mid A_j)\,\PP(A_j)}.

The numerator uses the definition of conditional probability, and the denominator is $\PP(B)$ expanded by the law of total probability.

Example 2.2 (False positive quiz – testing for a rare disease)

A medical test for a rare disease has the following properties:

If a person has the disease, the test is positive with probability $0.95$ (sensitivity).
If a person does not have the disease, the test is negative with probability $0.95$ (specificity), i.e. false positive rate $= 0.05$ .
The prior prevalence is $\PP(A) = 0.001$ (1 in 1000 people have the disease).

A person tests positive. What is $\PP(\text{disease} \mid \text{positive})$ ?

Define $A = \{\text{has disease}\}$ , $B = \{\text{tests positive}\}$ . By Bayes’ rule:

\PP(A \mid B) = \frac{\PP(B \mid A)\,\PP(A)}{\PP(B \mid A)\,\PP(A) + \PP(B \mid A^c)\,\PP(A^c)} = \frac{(0.95)(0.001)}{(0.95)(0.001) + (0.05)(0.999)} \approx 0.0187.

Despite the test being 95% accurate, a positive result only indicates a $\approx 1.87\%$ chance of actually having the disease. This is because the disease is so rare that the false positives from healthy people vastly outnumber the true positives.

Sensitivity to the prior. If the prior were $\PP(A) = 0.01$ instead ( $10\times$ more common), then $\PP(A \mid B) \approx 16\%$ — still low, but dramatically higher. The base rate (prior) has an enormous effect on the posterior.

Independence

Definition 2.2 (Independence of two events)

Two events $A$ and $B$ are independent if the occurrence of one provides no information about the occurrence of the other. Formally:

\PP(A \mid B) = \PP(A), \qquad \text{or equivalently,} \qquad \PP(A \cap B) = \PP(A)\,\PP(B).

The second form says the joint probability equals the product of the marginals.

Key remarks on independence:

Disjoint $\ne$ independent. $A, B$ disjoint means $\PP(A \cap B) = 0$ . For this to equal $\PP(A)\PP(B)$ , we need $\PP(A) = 0$ or $\PP(B) = 0$ . So disjoint events with positive probability are dependent, not independent.
Base outcomes are always dependent. Individual outcomes $\{\omega\}$ in any finite $\Omega$ are always dependent (knowing one occurred rules out the others).

Example 2.3 (Dice: independence depends on the conditioning event)

Roll two fair dice. Let $A = \{\text{1st die is 6}\}$ , $B = \{\text{sum is 7}\}$ , $C = \{\text{sum is 11}\}$ .

Are $A$ and $B$ independent? $\PP(A \mid B) = 1/6 = \PP(A)$ , so yes, $A \perp B$ .

Are $A$ and $C$ independent? $\PP(A \mid C) = 1/2 \ne 1/6 = \PP(A)$ , so no, $A$ and $C$ are dependent. Intuitively, knowing the sum is 11 makes it more likely that the first die was a 6 (since $(6,5)$ is one of only two outcomes).

Conditional Independence

Definition 2.3 (Conditional independence)

Events $A$ and $B$ are conditionally independent given $C$ if

\PP(A \cap B \mid C) = \PP(A \mid C)\,\PP(B \mid C).

Independence and conditional independence are distinct concepts:

Dependent events can become conditionally independent.
Independent events can become conditionally dependent.

Example 2.4 (Two indistinguishable coins)

You have two coins: one is double-headed, the other is double-tailed. You pick one at random and flip it twice. Let $H_i = \{\text{flip } i \text{ is heads}\}$ , $i=1,2$ .

Q1: Are $H_1, H_2$ (unconditionally) independent? No. $\PP(H_1 \mid H_2) = 1 \ne 1/2 = \PP(H_1)$ . If the second flip is heads, you certainly picked the double-headed coin, so the first flip is heads with certainty.

Q2: Conditioning on the coin, are they independent? Let $A = \{\text{picked the double-headed coin}\}$ . Then $\PP(H_1 \cap H_2 \mid A) = \PP(H_1 \mid A)\,\PP(H_2 \mid H_1 \cap A) = 1 \cdot 1 = 1$ , and $\PP(H_1 \mid A)\,\PP(H_2 \mid A) = 1 \cdot 1 = 1$ . These are equal, so $H_1 \perp H_2 \mid A$ . Once we know which coin was picked, each flip is determined and independent.

Example 2.5 (Fair coin – independent but conditionally dependent)

Toss a fair coin twice. Let $H_1 = \{\text{1st toss is H}\}$ , $H_2 = \{\text{2nd toss is H}\}$ , and $D = \{\text{1st toss} \ne \text{2nd toss}\}$ .

Q1: $H_1 \perp H_2$ ? Yes, by independence of fair coin flips.

Q2: $H_1 \perp H_2 \mid D$ ? No. $\PP(H_1 \mid D)\,\PP(H_2 \mid D) = \tfrac{1}{2} \cdot \tfrac{1}{2} = \tfrac{1}{4}$ . But $\PP(H_1 \cap H_2 \mid D) = 0$ , since both flips being heads contradicts $D$ (the flips must differ). Since $0 \ne 1/4$ , the events are conditionally dependent given $D$ .

Joint Independence vs. Pairwise Independence

Definition 2.4 (Independence of a collection)

Events $\{A_i\}_{i \in S}$ are jointly (mutually) independent if for every subset $T \subseteq S$ ,

\PP\!\Bigl(\bigcap_{i \in T} A_i\Bigr) = \prod_{i \in T} \PP(A_i).

Pairwise independence only requires this for all pairs $|T| = 2$ .

Pairwise independence $\nRightarrow$ joint independence. Consider the fair coin example: $H_1, H_2, D$ are pairwise independent (check all three pairs), but they are not jointly independent because $\PP(H_1 \cap H_2 \cap D) = 0 \ne \PP(H_1)\PP(H_2)\PP(D) = 1/8$ .

Renewal: Rolling a 6 Before a 7

Example 2.6 (Roll a sum of 6 before a sum of 7)

Roll two fair dice independently and repeatedly. Let $E = \{\text{a sum of 6 appears before a sum of 7}\}$ .

The outcomes that sum to 6 are $\{(1,5),(2,4),(3,3),(4,2),(5,1)\}$ , so $\PP(S) = 5/36$ where $S = \{\text{1st roll sums to 6}\}$ . Similarly, $\PP(T) = 6/36$ where $T = \{\text{1st roll sums to 7}\}$ .

By the law of total probability, conditioning on the first roll:

\PP(E) = \PP(E \mid S)\,\PP(S) + \PP(E \mid T)\,\PP(T) + \PP(E \mid (S \cup T)^c)\,\PP((S \cup T)^c).

We have $\PP(E \mid S) = 1$ , $\PP(E \mid T) = 0$ , and $\PP(E \mid (S \cup T)^c) = \PP(E)$ by the renewal/memoryless property (each roll is independent, so if neither 6 nor 7 appears, we restart). Thus

\PP(E) = 1 \cdot \frac{5}{36} + 0 \cdot \frac{6}{36} + \PP(E) \cdot \frac{25}{36}.

Solving: $\PP(E)(1 - 25/36) = 5/36$ , so $\PP(E)(11/36) = 5/36$ , giving

\PP(E) = \frac{5}{11}.

Lecture 3: Discrete Random Variables and Expectation

EECS 126 (UC Berkeley) · Spring 2026 · January 27, 2026

We transition from events to random variables — functions that assign numerical values to outcomes — and develop the key tools of expectation, linearity, LOTUS, and variance.

Random Variables and PMFs

Definition 3.1 (Random variable)

A (discrete) random variable is a function $X : \Omega \to \RR$ that assigns a real number to each outcome $\omega \in \Omega$ . We write $\PP(X = x)$ for the probability of the event $\{\omega \in \Omega : X(\omega) = x\}$ .

Definition 3.2 (Probability mass function)

The probability mass function (PMF) of a discrete random variable $X$ is the function $p_X : \RR \to [0,1]$ defined by

p_X(x) = \PP(X = x).

The PMF satisfies $\sum_{x \in \mathcal{X}} p_X(x) = 1$ , where $\mathcal{X}$ denotes the set of all values taken by $X$ (the support).

Example 3.1 (Functions of random variables)

Let $X$ be a random variable with $p_X(x) = 1/5$ for $x \in \{-2, -1, 0, 1, 2\}$ . Define $Y = |X|$ . Then $Y$ takes values in $\{0, 1, 2\}$ and

p_Y(y) = \sum_{\{x\,:\, |x| = y\}} p_X(x) = \begin{cases} 1/5 & \text{if } y = 0, \\ 2/5 & \text{if } y = 1, \\ 2/5 & \text{if } y = 2. \end{cases}

In general, if $Y = g(X)$ , then $p_Y(y) = \sum_{\{x : g(x) = y\}} p_X(x)$ .

Popular Discrete Distributions

Definition 3.3 (Bernoulli distribution)

$X \sim \Bern(p)$ : a single coin toss.

\PP(X = k) = \begin{cases} 1-p & k = 0, \\ p & k = 1. \end{cases}

Definition 3.4 (Binomial distribution)

$X \sim \Bin(n, p)$ : number of successes in $n$ independent Bernoulli $(p)$ trials.

\PP(X = k) = \binom{n}{k} p^k (1-p)^{n-k}, \qquad k = 0, 1, \ldots, n.

Note: $\sum_{k=0}^n \binom{n}{k} p^k(1-p)^{n-k} = (p + (1-p))^n = 1$ by the binomial theorem, confirming this is a valid PMF.

Definition 3.5 (Geometric distribution)

$X \sim \Geom(p)$ : number of trials until the first success.

\PP(X = k) = (1-p)^{k-1} p, \qquad k = 1, 2, 3, \ldots

The geometric distribution is the discrete analog of “waiting time.”

Definition 3.6 (Poisson distribution)

$X \sim \Pois(\lambda)$ : models the number of arrivals in a fixed interval.

\PP(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}, \qquad k = 0, 1, 2, \ldots

The Poisson is the limiting distribution of $\Bin(n, \lambda/n)$ as $n \to \infty$ , and arises naturally in modeling rare events and arrival processes.

Example 3.2 (Chess match – a PMF computation)*

Alice and Bob play a match: the first to win a game wins the match. If there are 10 consecutive draws, the match is declared drawn. Each game independently: $\PP(\text{A wins}) = 0.3$ , $\PP(\text{B wins}) = 0.4$ , $\PP(\text{draw}) = 0.3$ .

Let $L$ be the duration (number of games). Then for $\ell = 1, 2, \ldots, 9$ :

p_L(\ell) = (0.3)^{\ell - 1}(1 - 0.3) = (0.3)^{\ell-1}(0.7),

since the first $\ell - 1$ games must be draws and the $\ell$ -th decisive. For $\ell = 10$ : $p_L(10) = (0.3)^9$ (all 9 games drawn, match ends).

$\PP(\text{Alice wins match}) = \sum_{\ell=1}^{9}(0.3)^{\ell-1}(0.3) = 0.3 \sum_{k=0}^{8}(0.3)^k$ . Using $\sum_{k=0}^{\infty} r^k = 1/(1-r)$ for $|r|<1$ , in the limit of no cap on draws: $\PP(\text{Alice wins}) = 0.3/(1-0.3) = 3/7$ .

Expectation

Definition 3.7 (Expected value)

The expected value (or mean) of a discrete random variable $X$ is

\EE[X] = \sum_{\omega \in \Omega} X(\omega)\,\PP(\{\omega\}) = \sum_{x \in \mathcal{X}} x \cdot \PP(X = x),

provided the sum converges absolutely (i.e. $\sum |x|\,\PP(X=x) < \infty$ ).

Theorem 3.1 (Linearity of expectation)

For any random variables $X, Y$ (not necessarily independent) and constant $c$ :

\EE[X + Y] = \EE[X] + \EE[Y], \qquad \EE[cX] = c\,\EE[X].

No independence assumption is needed. This is the single most useful property of expectation.

Definition 3.8 (Indicator random variable)

For an event $A$ , the indicator $\mathbf{1}_A$ is

\mathbf{1}_A(\omega) = \begin{cases} 1 & \omega \in A, \\ 0 & \omega \notin A. \end{cases}

Observe that $\mathbf{1}_A \sim \Bern(\PP(A))$ , so $\EE[\mathbf{1}_A] = \PP(A)$ .

Example 3.3 (Seats on a plane)

A full flight has $n$ passengers with assigned seats, but every passenger ignores their assignment and sits in a uniformly random seat. What is $\EE[\text{number who sit in their own seat}]$ ?

Let $X_i = \mathbf{1}\{\text{passenger } i \text{ sits in their own seat}\}$ . Then $X = X_1 + X_2 + \cdots + X_n$ . The $X_i$ are not independent (if $n = 2$ and passenger 1 is in their seat, passenger 2 must be too). But linearity of expectation does not require independence:

\EE[X] = \sum_{i=1}^n \EE[X_i] = \sum_{i=1}^n \PP(\text{passenger } i \text{ is in their seat}) = n \cdot \frac{1}{n} = 1.

Regardless of the number of passengers, on average exactly 1 person sits in their own seat.

LOTUS (Law of the Unconscious Statistician)

Theorem 3.2 (LOTUS)

If $X$ is a discrete random variable and $g : \RR \to \RR$ is any function, then

\EE[g(X)] = \sum_{x \in \mathcal{X}} g(x)\,\PP(X = x).

More generally, for a function of several random variables:

\EE[f(X_1, \ldots, X_n)] = \sum_{x_1, \ldots, x_n} f(x_1, \ldots, x_n)\, p_{X_1, \ldots, X_n}(x_1, \ldots, x_n).

The name reflects that one can compute $\EE[g(X)]$ directly from the distribution of $X$ , without first finding the distribution of $Y = g(X)$ .

Independence of Random Variables

Definition 3.9 (Independent random variables)

Random variables $X$ and $Y$ are independent (written $X \perp Y$ ) if their joint distribution factors into the product of the marginals:

\PP(X = x,\, Y = y) = \PP(X = x)\,\PP(Y = y) \quad \text{for all } x, y.

Theorem 3.3 (Expectation of a product of independent RVs)

If $X \perp Y$ , then

\EE[XY] = \EE[X]\,\EE[Y].

Proof.

By LOTUS and independence:

\begin{aligned} \EE[XY] &= \sum_{x,y} xy\, p_{X,Y}(x,y) = \sum_{x,y} xy\, p_X(x)\, p_Y(y) \\ &= \Bigl(\sum_x x\, p_X(x)\Bigr)\Bigl(\sum_y y\, p_Y(y)\Bigr) = \EE[X]\,\EE[Y]. \end{aligned}

The key step is factoring the joint PMF into marginals (by independence), which allows separating the double sum into a product of single sums.

The converse is false. $\EE[XY] = \EE[X]\EE[Y]$ does not imply $X \perp Y$ . The condition $\EE[XY] = \EE[X]\EE[Y]$ is called uncorrelation, which is strictly weaker than independence. As an exercise, construct dependent random variables that are uncorrelated.

Tail Sum Formula

Theorem 3.4 (Tail sum formula)

Let $X$ be a random variable taking values in $\NN = \{0, 1, 2, \ldots\}$ . Then

\EE[X] = \sum_{k=1}^{\infty} \PP(X \ge k).

Proof.

Starting from the definition of expectation:

\begin{aligned} \EE[X] &= \sum_{k=1}^{\infty} k\,\PP(X = k) = \sum_{k=1}^{\infty} \sum_{j=1}^{k} 1 \cdot \PP(X = k) \\ &= \sum_{j=1}^{\infty} \sum_{k=j}^{\infty} \PP(X = k) = \sum_{j=1}^{\infty} \PP(X \ge j). \end{aligned}

The key step is writing $k = \sum_{j=1}^k 1$ to introduce a double sum, then swapping the order of summation (Tonelli’s theorem ensures this is valid for nonnegative terms). Geometrically, this corresponds to “counting vertically then horizontally”: instead of weighting each column $k$ by its height $k$ , we sum across rows $j = 1, 2, \ldots$ , each contributing $\PP(X \ge j)$ .

Intuition for the tail sum. Think of $\EE[X]$ as the total area under the “staircase” of the survival function $\PP(X \ge k)$ . Each unit step at height $k$ contributes $\PP(X \ge k)$ to the total. This is the same idea as computing the area under a curve by integrating horizontally rather than vertically.

Variance

Definition 3.10 (Variance and standard deviation)

The variance of $X$ measures the spread of $X$ around its mean:

\Var(X) \triangleq \sigma_X^2 \triangleq \EE\bigl[(X - \EE[X])^2\bigr].

The standard deviation is $\sigma_X = \sqrt{\Var(X)}$ .

An equivalent computational formula (often easier to use):

\Var(X) = \EE[X^2] - (\EE[X])^2.

Derivation of the computational formula.

Expand the square using linearity:

\begin{aligned} \Var(X) &= \EE[(X - \EE[X])^2] = \EE[X^2 - 2X\,\EE[X] + (\EE[X])^2] \\ &= \EE[X^2] - 2\,\EE[X]\,\EE[X] + (\EE[X])^2 = \EE[X^2] - (\EE[X])^2. \end{aligned}

Note that $\EE[X]$ is a constant, so $\EE[2X \cdot \EE[X]] = 2\,\EE[X] \cdot \EE[X]$ and $\EE[(\EE[X])^2] = (\EE[X])^2$ .

Theorem 3.5 (Scaling property of variance)

For constants $a, c \in \RR$ :

\Var(aX + c) = a^2\,\Var(X).

Adding a constant shifts the mean but does not change the spread. Scaling by $a$ scales the variance by $a^2$ (and the standard deviation by $|a|$ ).

Proof.

Let $\mu = \EE[X]$ . Then $\EE[aX + c] = a\mu + c$ , so

\Var(aX + c) = \EE[(aX + c - a\mu - c)^2] = \EE[(a(X - \mu))^2] = a^2\,\EE[(X - \mu)^2] = a^2\,\Var(X).

Coming up next lecture: covariance $\Cov(X, Y)$ , variance of sums, and the beginning of continuous random variables.

Lecture 4: Variance, Covariance, and Common Discrete Distributions

EECS 126 (UC Berkeley) · Spring 2026 · January 29, 2026

Reading: Bertsekas & Tsitsiklis, Ch. 2; §4.2–4.3. Ch. 3 next week.

These are my personal notes from Professor Kannan Ramchandran’s EECS 126 (Probability and Random Processes) course at UC Berkeley.

Last lecture we defined expectation and variance for a single random variable. In this lecture we study how variance behaves for sums of random variables, which leads to covariance and the correlation coefficient. We then compute the mean and variance of the four workhorse discrete distributions (Bernoulli, binomial, geometric, Poisson), apply these to the coupon collector problem, prove the memoryless property of the geometric distribution, study how Poisson random variables merge and split, and finally introduce conditional expectation.

Variance of Sums and Covariance

Recall from Lecture 3 the two forms of the variance,

\Var(X) = \EE\bigl[(X - \EE[X])^2\bigr] = \EE[X^2] - (\EE[X])^2,

and the scaling property

\Var(aX + c) = a^2\,\Var(X).

Shifts by a constant $c$ are not recorded in variance calculations: shifting every outcome by $c$ moves the mean by $c$ as well, so the deviations $X - \EE[X]$ are unchanged.

The natural next question is: what is the variance of a sum $X + Y$ ? Unlike the mean, the answer depends on how $X$ and $Y$ interact.

Theorem 4.1 (Variance of a sum)

For any random variables $X, Y$ with finite variances,

\Var(X + Y) = \Var(X) + \Var(Y) + 2\bigl(\EE[XY] - \EE[X]\,\EE[Y]\bigr). \tag{4.3}

Proof.

\begin{aligned} \Var(X+Y) &= \EE\bigl[(X+Y)^2\bigr] - \bigl(\EE[X+Y]\bigr)^2 \\ &= \EE\bigl[X^2 + 2XY + Y^2\bigr] - \bigl(\EE[X] + \EE[Y]\bigr)^2 \\ &= \EE[X^2] + 2\,\EE[XY] + \EE[Y^2] - (\EE[X])^2 - 2\,\EE[X]\,\EE[Y] - (\EE[Y])^2 \\ &= \underbrace{\EE[X^2] - (\EE[X])^2}_{\Var(X)} \;+\; \underbrace{\EE[Y^2] - (\EE[Y])^2}_{\Var(Y)} \;+\; 2\,\EE[XY] - 2\,\EE[X]\,\EE[Y] \\ &= \Var(X) + \Var(Y) + 2\bigl(\EE[XY] - \EE[X]\,\EE[Y]\bigr). \end{aligned}

No independence was used. The derivation above holds for any $X$ and $Y$ . Independence only enters at the very end, where it makes the cross term vanish (Corollary 4.1 below).

The cross term in (4.3) is important enough to get its own name.

Definition 4.1 (Covariance)

The covariance of random variables $X$ and $Y$ is

\Cov(X, Y) \triangleq \EE\bigl[(X - \EE[X])(Y - \EE[Y])\bigr] = \EE[XY] - \EE[X]\,\EE[Y].

In particular, $\Cov(X, X) = \Var(X)$ . With this notation,

\Var(X + Y) = \Var(X) + \Var(Y) + 2\,\Cov(X, Y).

Equivalence of the two forms.

Write $\mu_X = \EE[X]$ and $\mu_Y = \EE[Y]$ , which are constants. Expanding the product and using linearity,

\begin{aligned} \EE\bigl[(X - \mu_X)(Y - \mu_Y)\bigr] &= \EE\bigl[XY - \mu_Y X - \mu_X Y + \mu_X\mu_Y\bigr] \\ &= \EE[XY] - \mu_Y\,\EE[X] - \mu_X\,\EE[Y] + \mu_X\mu_Y \\ &= \EE[XY] - \mu_Y\mu_X - \mu_X\mu_Y + \mu_X\mu_Y \\ &= \EE[XY] - \mu_X\mu_Y. \end{aligned}

Interpreting the sign. $\Cov(X,Y) > 0$ means $X$ and $Y$ tend to be above their means together (and below their means together). $\Cov(X,Y) < 0$ means when one is above its mean, the other tends to be below. For a sum, positive covariance inflates the spread and negative covariance dampens it.

Corollary 4.1 (Independent summands)

If $X \perp Y$ , then $\Cov(X, Y) = 0$ , and therefore

\Var(X + Y) = \Var(X) + \Var(Y).

Proof.

In Lecture 3 we proved that $X \perp Y$ implies $\EE[XY] = \EE[X]\,\EE[Y]$ . Hence $\Cov(X,Y) = \EE[XY] - \EE[X]\,\EE[Y] = 0$ , and the cross term in (4.3) vanishes.

Corollary 4.2 (Variance of a sum of many random variables)

For random variables $X_1, \ldots, X_n$ with finite variances,

\Var\Bigl(\sum_{i=1}^n X_i\Bigr) = \sum_{i=1}^n \Var(X_i) + 2\sum_{i<j} \Cov(X_i, X_j).

In particular, if the $X_i$ are independent (in fact, pairwise uncorrelated is enough), then

\Var\Bigl(\sum_{i=1}^n X_i\Bigr) = \sum_{i=1}^n \Var(X_i).

Proof.

Let $\mu_i = \EE[X_i]$ . Since $\EE[\sum_i X_i] = \sum_i \mu_i$ , the centered sum is $\sum_i (X_i - \mu_i)$ . Squaring a sum gives a double sum over all pairs $(i, j)$ :

\begin{aligned} \Var\Bigl(\sum_{i} X_i\Bigr) &= \EE\Bigl[\Bigl(\sum_{i} (X_i - \mu_i)\Bigr)^2\Bigr] = \EE\Bigl[\sum_{i}\sum_{j} (X_i - \mu_i)(X_j - \mu_j)\Bigr] \\ &= \sum_{i}\sum_{j} \Cov(X_i, X_j). \end{aligned}

The diagonal terms $i = j$ give $\Cov(X_i, X_i) = \Var(X_i)$ . The off-diagonal terms come in symmetric pairs, $\Cov(X_i, X_j) = \Cov(X_j, X_i)$ , so they contribute $2\sum_{i<j}\Cov(X_i,X_j)$ . If every pair is uncorrelated, all off-diagonal terms are zero.

Uncorrelated does not imply independent. Covariance only detects linear dependence. For example, let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$ . Then $\EE[X] = 0$ and $\EE[XY] = \EE[X^3] = \tfrac13(-1) + \tfrac13(0) + \tfrac13(1) = 0$ , so $\Cov(X, Y) = 0 - 0 \cdot \EE[Y] = 0$ . But $Y$ is completely determined by $X$ : for instance $\PP(Y = 0 \mid X = 0) = 1 \ne \tfrac13 = \PP(Y = 0)$ . This was a question Ramchandran asked us at end of previous lecture.

The Correlation Coefficient

Covariance has units (units of $X$ times units of $Y$ ), so its size alone does not tell us how strongly two variables are related. Normalizing by the standard deviations removes the units.

Definition 4.2 (Correlation coefficient)

For random variables $X, Y$ with $\Var(X) > 0$ and $\Var(Y) > 0$ , the correlation coefficient is

\rho(X, Y) \triangleq \frac{\Cov(X, Y)}{\sqrt{\Var(X)}\,\sqrt{\Var(Y)}}.

Ramchandran’s vector analogy. Think of random variables as vectors, with “dot product” $\langle U, V\rangle = \EE[UV]$ and “length” $\sqrt{\EE[U^2]}$ . For the centered variables $U = X - \EE[X]$ and $V = Y - \EE[Y]$ :

the dot product is $\EE[UV] = \Cov(X, Y)$ ;
the lengths are $\sqrt{\EE[U^2]} = \sqrt{\Var(X)}$ and $\sqrt{\EE[V^2]} = \sqrt{\Var(Y)}$ .

So $\rho$ plays the role of $\cos\theta = \dfrac{u \cdot v}{|u|\,|v|}$ , the cosine of the angle between two vectors. For ordinary vectors, the Cauchy–Schwarz inequality $|u \cdot v| \le |u|\,|v|$ forces $-1 \le \cos\theta \le 1$ . The same inequality, proved for random variables below, forces $-1 \le \rho \le 1$ .

Lemma 4.1 (Cauchy–Schwarz inequality for random variables)

For any random variables $U, V$ with $\EE[U^2] < \infty$ and $\EE[V^2] < \infty$ ,

\bigl(\EE[UV]\bigr)^2 \le \EE[U^2]\,\EE[V^2].

Proof.

Case 1: $\EE[V^2] = 0$ . Since $V^2 \ge 0$ and its mean is $0$ , we must have $V = 0$ with probability $1$ . Then $UV = 0$ with probability $1$ , so $\EE[UV] = 0$ and both sides equal $0$ .

Case 2: $\EE[V^2] > 0$ . For every real number $t$ , the random variable $(U - tV)^2$ is nonnegative, so its expectation is nonnegative. Expanding with linearity:

0 \le \EE\bigl[(U - tV)^2\bigr] = \EE[U^2] - 2t\,\EE[UV] + t^2\,\EE[V^2].

This holds for every $t$ , so choose the $t$ that minimizes the right-hand side, namely $t^\star = \EE[UV]/\EE[V^2]$ . Substituting,

\begin{aligned} 0 &\le \EE[U^2] - 2\,\frac{\EE[UV]}{\EE[V^2]}\,\EE[UV] + \frac{(\EE[UV])^2}{(\EE[V^2])^2}\,\EE[V^2] \\ &= \EE[U^2] - 2\,\frac{(\EE[UV])^2}{\EE[V^2]} + \frac{(\EE[UV])^2}{\EE[V^2]} \\ &= \EE[U^2] - \frac{(\EE[UV])^2}{\EE[V^2]}. \end{aligned}

Multiplying both sides by $\EE[V^2] > 0$ gives $(\EE[UV])^2 \le \EE[U^2]\,\EE[V^2]$ .

Theorem 4.2 (Range of the correlation coefficient)

Whenever $\rho(X, Y)$ is defined,

-1 \le \rho(X, Y) \le 1.

Proof.

Apply Cauchy–Schwarz to the centered variables $U = X - \EE[X]$ and $V = Y - \EE[Y]$ . By definition, $\EE[UV] = \Cov(X,Y)$ , $\EE[U^2] = \Var(X)$ , and $\EE[V^2] = \Var(Y)$ , so

\bigl(\Cov(X,Y)\bigr)^2 \le \Var(X)\,\Var(Y).

Taking square roots, $|\Cov(X,Y)| \le \sqrt{\Var(X)}\sqrt{\Var(Y)}$ . Dividing by the (positive) right-hand side gives $|\rho(X,Y)| \le 1$ .

Extreme and zero values. $|\rho| = 1$ exactly when equality holds in Cauchy–Schwarz, which happens when $U - t^\star V = 0$ with probability $1$ , i.e. when $Y$ is an exact linear function $Y = aX + b$ of $X$ (with $\rho = +1$ if $a > 0$ and $\rho = -1$ if $a < 0$ ). At the other end, $\rho = 0$ means uncorrelated, which, as we just saw, is weaker than independent.

Mean and Variance of Common Distributions

We now compute $\EE[X]$ and $\Var(X)$ for the four distributions introduced in Lecture 3. A recurring theme: the direct sums are often messy, and a better representation of the random variable makes the computation easy.

Bernoulli

Theorem 4.3 (Bernoulli mean and variance)

If $X \sim \Bern(p)$ , then

\EE[X] = p, \qquad \Var(X) = p(1-p).

Proof.

$X$ takes only the values $0$ and $1$ , with $\PP(X=1) = p$ and $\PP(X=0) = 1-p$ .

\EE[X] = 1 \cdot \PP(X = 1) + 0 \cdot \PP(X = 0) = 1 \cdot p + 0 = p.

For the second moment, by LOTUS,

\EE[X^2] = \sum_{x} x^2\,\PP(X = x) = 1^2 \cdot \PP(X = 1) + 0^2 \cdot \PP(X = 0) = p.

Therefore

\Var(X) = \EE[X^2] - (\EE[X])^2 = p - p^2 = p(1-p).

Binomial

Theorem 4.4 (Binomial mean and variance)

If $X \sim \Bin(n, p)$ , then

\EE[X] = np, \qquad \Var(X) = np(1-p).

The direct approach is

\EE[X] = \sum_{x=0}^{n} x \cdot \PP(X = x) = \sum_{x=0}^{n} x \binom{n}{x} p^x (1-p)^{n-x},

which is doable but unpleasant. Instead, think of the binomial as a sum of $n$ Bernoulli trials.

Proof.

Let $X_i = \mathbf{1}\{\text{trial } i \text{ is a success}\}$ for $i = 1, \ldots, n$ . Each $X_i \sim \Bern(p)$ , the $X_i$ are independent (the trials are independent), and the total number of successes is

X = X_1 + X_2 + \cdots + X_n.

Mean. By linearity of expectation (which needs no independence),

\EE[X] = \sum_{i=1}^{n} \EE[X_i] = \sum_{i=1}^{n} p = np.

Variance. Because the $X_i$ are independent, Corollary 4.2 lets us add the variances:

\Var(X) = \sum_{i=1}^{n} \Var(X_i) = \sum_{i=1}^{n} p(1-p) = np(1-p).

Applying direct sum gives us the same result: Using the identity $x\binom{n}{x} = n\binom{n-1}{x-1}$ for $x \ge 1$ (the $x = 0$ term is zero),

\begin{aligned} \EE[X] &= \sum_{x=1}^{n} n\binom{n-1}{x-1} p^x (1-p)^{n-x} = np \sum_{x=1}^{n} \binom{n-1}{x-1} p^{x-1}(1-p)^{(n-1)-(x-1)} \\ &= np \sum_{j=0}^{n-1} \binom{n-1}{j} p^{j}(1-p)^{(n-1)-j} = np\,\bigl(p + (1-p)\bigr)^{n-1} = np, \end{aligned}

where we substituted $j = x - 1$ and used the binomial theorem. The indicator method reaches the same answer with far less algebra.

Geometric

Recall that $X \sim \Geom(p)$ counts the number of independent trials up to and including the first success:

p_X(k) = \PP(X = k) = (1-p)^{k-1}\,p, \qquad k = 1, 2, 3, \ldots

Reading the PMF. Here $k$ marks the trial on which the first Success occurs ( $k = S$ ). So $p_X(k)$ is the probability that the first success happens on trial $k$ : the first $k - 1$ trials are failures (probability $(1-p)^{k-1}$ ) and trial $k$ is a success (probability $p$ ). Note that $p_X(k)$ is not the probability of $k$ successes; a geometric random variable always records exactly one success, and $k$ counts how long we waited for it.

Definition 4.3 (Survival function)

For an integer-valued random variable $X$ , the survival function (or tail) is $k \mapsto \PP(X > k)$ . For $X \sim \Geom(p)$ ,

\PP(X > k) = (1-p)^k, \qquad k = 0, 1, 2, \ldots \tag{4.27}

Proof.

The event $\{X > k\}$ means “no success in the first $k$ trials,” i.e. we tried $k$ times and failed every time. The trials are independent and each fails with probability $1-p$ , so $\PP(X > k) = \underbrace{(1-p)\cdots(1-p)}_{k \text{ times}} = (1-p)^k$ .

Theorem 4.5 (Geometric mean)

If $X \sim \Geom(p)$ with $0 < p \le 1$ , then

\EE[X] = \frac{1}{p}.

The direct sum $\EE[X] = \sum_{k=1}^\infty k(1-p)^{k-1}p$ requires a trick. A better method is the tail sum formula from Lecture 3, which applies because $X$ takes values in the nonnegative integers.

Proof.

By the tail sum formula,

\EE[X] = \sum_{k=1}^{\infty} \PP(X \ge k).

For integer-valued $X$ , $\{X \ge k\} = \{X > k - 1\}$ . Substituting $j = k - 1$ (so $j$ runs from $0$ to $\infty$ ),

\EE[X] = \sum_{k=1}^{\infty} \PP(X > k-1) = \sum_{j=0}^{\infty} \PP(X > j).

Now plug in the survival function (4.27):

\EE[X] = \sum_{j=0}^{\infty} (1-p)^j.

This is a geometric series $\sum_{j \ge 0} x^j$ with ratio $x = 1 - p$ . Since $0 < p \le 1$ we have $|x| < 1$ , so the series converges to $1/(1-x)$ :

\EE[X] = \frac{1}{1 - (1-p)} = \frac{1}{p}.

Exam note: tail sum formula will be on exam. For a random variable $X$ taking values in $\{0, 1, 2, \ldots\}$ ,

\EE[X] = \sum_{k=1}^{\infty} \PP(X \ge k) = \sum_{k=0}^{\infty} \PP(X > k).

It expresses the mean as the sum of the tail probabilities. It follows from writing $X = \sum_{k=1}^{\infty} \mathbf{1}\{X \ge k\}$ and taking expectations of both sides.

The direct sum, for comparison. Differentiating the geometric series $\sum_{k\ge0} x^k = \frac{1}{1-x}$ term by term (valid for $|x|<1$ ) gives $\sum_{k\ge1} k x^{k-1} = \frac{1}{(1-x)^2}$ . With $x = 1-p$ ,

\EE[X] = p\sum_{k=1}^{\infty} k(1-p)^{k-1} = p \cdot \frac{1}{(1-(1-p))^2} = p \cdot \frac{1}{p^2} = \frac{1}{p}.

The variance of the geometric is derived in Section 4.5, using the memoryless property.

Poisson

Theorem 4.6 (Poisson mean and variance)

If $X \sim \Pois(\lambda)$ , so that $p_X(k) = \dfrac{e^{-\lambda}\lambda^k}{k!}$ for $k = 0, 1, 2, \ldots$ , then

\EE[X] = \lambda, \qquad \Var(X) = \lambda.

Both computations rely on the Taylor series of the exponential,

e^{x} = \sum_{i=0}^{\infty} \frac{x^i}{i!} = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots, \tag{4.36}

and on the cancellation $\dfrac{k}{k!} = \dfrac{1}{(k-1)!}$ for $k \ge 1$ .

Proof of the mean.

\begin{aligned} \EE[X] &= \sum_{k=0}^{\infty} k \cdot \frac{e^{-\lambda}\lambda^k}{k!} \\ &= \sum_{k=1}^{\infty} k \cdot \frac{e^{-\lambda}\lambda^k}{k!} && \text{(the } k = 0 \text{ term is } 0\text{)} \\ &= \sum_{k=1}^{\infty} \frac{e^{-\lambda}\lambda^k}{(k-1)!} && \text{(cancel } k \text{ against } k!\text{)} \\ &= \lambda e^{-\lambda} \sum_{k=1}^{\infty} \frac{\lambda^{k-1}}{(k-1)!} && \text{(factor out } \lambda e^{-\lambda}\text{)} \\ &= \lambda e^{-\lambda} \sum_{j=0}^{\infty} \frac{\lambda^{j}}{j!} && \text{(substitute } j = k - 1\text{)} \\ &= \lambda e^{-\lambda} e^{\lambda} = \lambda. && \text{(Taylor series (4.36))} \end{aligned}

For the variance, computing $\EE[X^2] = \sum k^2 \frac{e^{-\lambda}\lambda^k}{k!}$ directly is awkward because $k^2$ does not cancel cleanly against $k!$ . The fix is the factorial moment trick: since $X^2 = X(X-1) + X$ , linearity gives

\EE[X^2] = \EE[X(X-1)] + \EE[X], \tag{4.37}

and $k(k-1)$ cancels against $k!$ just as $k$ did.

Proof of the variance.

Step 1: the factorial moment.

\begin{aligned} \EE[X(X-1)] &= \sum_{k=0}^{\infty} k(k-1)\,\frac{e^{-\lambda}\lambda^k}{k!} \\ &= \sum_{k=2}^{\infty} k(k-1)\,\frac{e^{-\lambda}\lambda^k}{k!} && \text{(the } k = 0, 1 \text{ terms are } 0\text{)} \\ &= \sum_{k=2}^{\infty} \frac{e^{-\lambda}\lambda^k}{(k-2)!} && \text{(cancel } k(k-1) \text{ against } k!\text{)} \\ &= e^{-\lambda}\lambda^2 \sum_{k=2}^{\infty} \frac{\lambda^{k-2}}{(k-2)!} && \text{(factor out } e^{-\lambda}\lambda^2\text{)} \\ &= e^{-\lambda}\lambda^2 e^{\lambda} = \lambda^2. && \text{(same trick: substitute } j = k-2\text{)} \end{aligned}

Step 2: the second moment. By (4.37),

\EE[X^2] = \EE[X(X-1)] + \EE[X] = \lambda^2 + \lambda.

Step 3: the variance.

\Var(X) = \EE[X^2] - (\EE[X])^2 = (\lambda^2 + \lambda) - (\lambda)^2 = \lambda.

Mean equals variance (Poisson distribution). The Poisson is the only one of our four distributions whose mean and variance are equal. This is consistent with the Poisson being the limit of $\Bin(n, \lambda/n)$ : the binomial variance is $n \cdot \frac{\lambda}{n}\bigl(1 - \frac{\lambda}{n}\bigr) = \lambda\bigl(1 - \frac{\lambda}{n}\bigr) \to \lambda$ as $n \to \infty$ .

The Coupon Collector Problem

Example 4.1 (Coupon collector)

There are $N$ different coupons you want to collect, found inside cereal boxes. Each box contains exactly one coupon, whose type is uniformly random among the $N$ types and independent of all other boxes. You buy boxes until you have received all $N$ distinct coupons. What is the expected number of boxes you must buy?

Let $T$ be the total time (number of boxes) to collect all $N$ coupons. Break the process into $N$ stages. For $i = 1, \ldots, N$ , let

T_i = \#\text{ of boxes bought to obtain the } i\text{-th \textit{new} coupon,}

counting from right after the $(i-1)$ -th new coupon was obtained. Then the total time is the sum of the stage times,

T = \sum_{i=1}^{N} T_i.

Each stage is geometric. During stage $i$ we already hold $i - 1$ distinct coupons, so $N - (i-1) = N - i + 1$ types would be new. Each box is independently a “success” (a new type) with probability

p_i = \frac{N - i + 1}{N}, \qquad\text{so}\qquad T_i \sim \Geom\!\left(\frac{N - i + 1}{N}\right).

$i = 1$ : $T_1 \sim \Geom\bigl(\frac{N - 1 + 1}{N}\bigr) = \Geom(1)$ , so $T_1 = 1$ . The first box always gives a new coupon.
$i = 2$ : $T_2 \sim \Geom\bigl(\frac{N - 2 + 1}{N}\bigr) = \Geom\bigl(\frac{N-1}{N}\bigr)$ .
$i = N$ : $T_N \sim \Geom\bigl(\frac{1}{N}\bigr)$ . The last coupon is the hardest to find.

Expected total. Since $\EE[T_i] = 1/p_i = \frac{N}{N - i + 1}$ , linearity of expectation gives

\EE[T] = \EE\Bigl[\sum_{i=1}^{N} T_i\Bigr] = \sum_{i=1}^{N} \EE[T_i] = \sum_{i=1}^{N} \frac{N}{N - i + 1}.

Write out the terms, from $i = 1$ (denominator $N$ ) to $i = N$ (denominator $1$ ):

\begin{aligned} \EE[T] &= \frac{N}{N} + \frac{N}{N-1} + \frac{N}{N-2} + \cdots + \frac{N}{2} + \frac{N}{1} \\ &= N\left(\frac{1}{N} + \frac{1}{N-1} + \cdots + \frac{1}{2} + \frac{1}{1}\right) = N \sum_{j=1}^{N} \frac{1}{j}, \end{aligned}

where we substituted $j = N - i + 1$ (as $i$ runs from $1$ to $N$ , $j$ runs from $N$ down to $1$ ). The sum is the $N$ -th harmonic number $H_N = \sum_{j=1}^N \frac1j$ , so

\EE[T] = N\,H_N.

Lemma 4.2 (Size of the harmonic numbers)

For every $N \ge 1$ ,

\ln(N + 1) \le H_N \le 1 + \ln N,

and more precisely $H_N = \ln N + \gamma + o(1)$ , where $\gamma \approx 0.5772$ is the Euler–Mascheroni constant.

Proof of the bounds.

Compare each term $1/j$ with the area under the curve $1/x$ . Since $1/x$ is decreasing, on the interval $[j, j+1]$ we have $1/x \le 1/j$ , and on $[j-1, j]$ we have $1/x \ge 1/j$ . Hence

\int_j^{j+1}\frac{dx}{x} \;\le\; \frac{1}{j} \;\le\; \int_{j-1}^{j}\frac{dx}{x} \qquad (j \ge 2 \text{ for the right inequality}).

Summing the left inequality over $j = 1, \ldots, N$ gives $H_N \ge \int_1^{N+1}\frac{dx}{x} = \ln(N+1)$ . Summing the right inequality over $j = 2, \ldots, N$ and adding the $j=1$ term gives $H_N \le 1 + \int_1^{N}\frac{dx}{x} = 1 + \ln N$ . (The sharper statement with $\gamma$ is a classical result we take as given.)

Corollary 4.3 (Coupon collector asymptotics)

\EE[T] = N\,H_N \approx N \ln N + \gamma N \approx N\ln N.

So we need to buy about a factor of $\ln N$ more boxes than the $N$ coupons we are trying to collect.

Numbers and intuition. For $N = 50$ , $H_{50} \approx 4.499$ , so $\EE[T] \approx 225$ boxes, about $4.5\times$ the number of coupon types. Most of the cost comes at the end: the final coupon alone takes $\EE[T_N] = N$ boxes on average, while the entire first half of the collection takes fewer than $N$ boxes in total ( $\sum_{i \le N/2} \frac{N}{N-i+1} \le \frac{N}{2}\cdot 2 = N$ ). This $N\ln N$ behavior shows up whenever we need to “hit every bin” by random sampling: hashing, load balancing, and covering every node in a network.

The Memoryless Property

Theorem 4.7 (Memoryless property of the geometric)

Let $X \sim \Geom(p)$ with $0 < p < 1$ . For all integers $s, t \ge 0$ ,

\PP(X > s + t \mid X > s) = \PP(X > t).

In words: given that we have already failed $s$ times, the probability of failing $t$ more times is the same as if we were starting over from scratch.

Proof.

First note that $\PP(X > s) = (1-p)^s > 0$ , so the conditional probability is well defined. By the definition of conditional probability,

\PP(X > s + t \mid X > s) = \frac{\PP\bigl(\{X > s + t\} \cap \{X > s\}\bigr)}{\PP(X > s)}.

Since $s + t \ge s$ , the event $\{X > s + t\}$ is contained in $\{X > s\}$ (if we have $s+t$ failures, we certainly have $s$ failures), so the intersection is just $\{X > s + t\}$ :

\PP(X > s + t \mid X > s) = \frac{\PP(X > s + t)}{\PP(X > s)}.

Both numerator and denominator are values of the geometric survival function (4.27):

\frac{\PP(X > s + t)}{\PP(X > s)} = \frac{(1-p)^{s+t}}{(1-p)^{s}} = (1-p)^{t},

which is exactly the survival function of the geometric evaluated at $t$ , i.e. $\PP(X > t)$ .

Two remarks on memorylessness.

The remaining wait is again geometric. Given $X > s$ , the number of additional trials $X - s$ has the same distribution as $X$ : for $k \ge 1$ , $\PP(X - s = k \mid X > s) = \frac{\PP(X = s + k)}{\PP(X > s)} = \frac{(1-p)^{s+k-1}\,p}{(1-p)^{s}} = (1-p)^{k-1}\,p.$ This is the form we use in the derivations below.
Uniqueness. The geometric is the only memoryless distribution on $\{1, 2, 3, \ldots\}$ . Its continuous analog, the exponential distribution, will be the only memoryless continuous distribution.

To use memorylessness to recompute $\EE[X]$ , we need to split an expectation according to which of several events occurred. This is the expectation version of the law of total probability from Lecture 2.

Theorem 4.8 (Total expectation theorem)

Let $A_1, \ldots, A_n$ be a partition of $\Omega$ with $\PP(A_i) > 0$ for each $i$ , and define the conditional expectation given an event, $\EE[X \mid A_i] = \sum_x x\,\PP(X = x \mid A_i)$ . Then

\EE[X] = \sum_{i=1}^{n} \PP(A_i)\,\EE[X \mid A_i].

Proof.

For each value $x$ , the law of total probability gives $\PP(X = x) = \sum_i \PP(A_i)\,\PP(X = x \mid A_i)$ . Substituting into the definition of expectation and swapping the order of summation (justified by absolute convergence),

\begin{aligned} \EE[X] &= \sum_x x\,\PP(X = x) = \sum_x x \sum_i \PP(A_i)\,\PP(X = x \mid A_i) \\ &= \sum_i \PP(A_i) \sum_x x\,\PP(X = x \mid A_i) = \sum_i \PP(A_i)\,\EE[X \mid A_i]. \end{aligned}

Example 4.2 (Geometric mean via memorylessness)

Let $X \sim \Geom(p)$ . Condition on the outcome of the first trial, using the partition $\{X = 1\}$ (first trial succeeds) and $\{X > 1\}$ (first trial fails):

\EE[X] = \PP(X = 1)\,\EE[X \mid X = 1] + \PP(X > 1)\,\EE[X \mid X > 1].

$\PP(X = 1) = p$ (success on the first try), and then $X = 1$ exactly, so $\EE[X \mid X = 1] = 1$ .
$\PP(X > 1) = 1 - p$ . If the first trial failed, the geometric “tells us we start again”: by memorylessness the remaining wait $X - 1$ is again $\Geom(p)$ , and we have already spent one trial. So $\EE[X \mid X > 1] = \EE[(X - 1) + 1 \mid X > 1] = \EE[X] + 1.$

Substituting,

\begin{aligned} \EE[X] &= p \cdot 1 + (1-p)\bigl(\EE[X] + 1\bigr) \\ &= p + \EE[X] + 1 - p\,\EE[X] - p \\ &= \EE[X] + 1 - p\,\EE[X]. \end{aligned}

Subtracting $\EE[X]$ from both sides gives $0 = 1 - p\,\EE[X]$ , i.e. $p\,\EE[X] = 1$ , so

\EE[X] = \frac{1}{p},

matching the tail sum computation.

Two checks on this argument.

Finiteness. Subtracting $\EE[X]$ from both sides is only legitimate because $\EE[X] < \infty$ , which we already know from the tail sum. (If $\EE[X] = \infty$ , the equation $\infty = \infty + 1 - \infty$ says nothing.)
Sanity check the answer. Any correct formula must satisfy $\EE[X] \ge 1$ (you need at least one trial) and $\EE[X] \to \infty$ as $p \to 0$ . A sign slip in the algebra can produce $\EE[X] = \frac{1}{1+p}$ , which fails both tests, since it is less than $1$ for every $p > 0$ .

Theorem 4.9 (Geometric variance (supplementary))

If $X \sim \Geom(p)$ , then

\Var(X) = \frac{1 - p}{p^2}.

Proof.

We use the same first-trial conditioning, now on $\EE[X^2]$ (which is finite, since the tail of $X$ decays geometrically).

\EE[X^2] = \PP(X = 1)\,\EE[X^2 \mid X = 1] + \PP(X > 1)\,\EE[X^2 \mid X > 1].

Given $X = 1$ : $X^2 = 1$ , so $\EE[X^2 \mid X = 1] = 1$ .
Given $X > 1$ : by memorylessness, $X = 1 + X'$ where $X' \sim \Geom(p)$ has the same distribution as $X$ . Expanding the square, $\EE[X^2 \mid X > 1] = \EE\bigl[(1 + X')^2\bigr] = 1 + 2\,\EE[X'] + \EE[X'^2] = 1 + 2\,\EE[X] + \EE[X^2].$

Substituting, with $\EE[X] = 1/p$ :

\begin{aligned} \EE[X^2] &= p \cdot 1 + (1-p)\bigl(1 + 2\,\EE[X] + \EE[X^2]\bigr) \\ &= p + (1 - p) + 2(1-p)\,\EE[X] + (1-p)\,\EE[X^2] \\ &= 1 + \frac{2(1-p)}{p} + (1-p)\,\EE[X^2]. \end{aligned}

Move the $\EE[X^2]$ terms to the left:

\EE[X^2] - (1-p)\,\EE[X^2] = p\,\EE[X^2] = 1 + \frac{2(1-p)}{p} = \frac{p + 2 - 2p}{p} = \frac{2 - p}{p}.

Dividing by $p$ , $\EE[X^2] = \dfrac{2 - p}{p^2}$ . Finally,

\Var(X) = \EE[X^2] - (\EE[X])^2 = \frac{2 - p}{p^2} - \frac{1}{p^2} = \frac{1 - p}{p^2}.

Cross-check with the factorial moment trick. Differentiating $\sum_{k\ge0}x^k = \frac{1}{1-x}$ twice gives $\sum_{k\ge2} k(k-1)x^{k-2} = \frac{2}{(1-x)^3}$ . With $x = 1-p$ ,

\EE[X(X-1)] = \sum_{k=2}^{\infty} k(k-1)(1-p)^{k-1}p = p(1-p)\sum_{k=2}^{\infty} k(k-1)(1-p)^{k-2} = p(1-p)\cdot\frac{2}{p^3} = \frac{2(1-p)}{p^2},

so $\EE[X^2] = \EE[X(X-1)] + \EE[X] = \frac{2(1-p)}{p^2} + \frac{p}{p^2} = \frac{2 - p}{p^2}$ , in agreement.

Poisson Merging and Splitting

Poisson random variables model counts of arrivals (packets at a router, calls at a switchboard, customers at a queue). Two structural facts make them especially convenient: independent Poisson counts merge into a Poisson count, and randomly splitting a Poisson count produces independent Poisson counts.

Theorem 4.10 (Poisson merging (sum of independent Poissons))

Let $X \sim \Pois(\lambda)$ and $Y \sim \Pois(\mu)$ be independent. Then

X + Y \sim \Pois(\lambda + \mu),

with mean $\EE[X + Y] = \lambda + \mu$ .

Proof (supplementary).

Fix $n \in \{0, 1, 2, \ldots\}$ . Since $X, Y \ge 0$ , the event $\{X + Y = n\}$ splits into the disjoint events $\{X = k, Y = n - k\}$ for $k = 0, \ldots, n$ :

\begin{aligned} \PP(X + Y = n) &= \sum_{k=0}^{n} \PP(X = k,\, Y = n - k) \\ &= \sum_{k=0}^{n} \PP(X = k)\,\PP(Y = n - k) && \text{(independence)} \\ &= \sum_{k=0}^{n} \frac{e^{-\lambda}\lambda^k}{k!} \cdot \frac{e^{-\mu}\mu^{n-k}}{(n-k)!} && \text{(Poisson PMFs)} \\ &= e^{-(\lambda+\mu)} \sum_{k=0}^{n} \frac{\lambda^k \mu^{n-k}}{k!\,(n-k)!} && \text{(collect exponentials)} \\ &= \frac{e^{-(\lambda+\mu)}}{n!} \sum_{k=0}^{n} \frac{n!}{k!\,(n-k)!}\,\lambda^k \mu^{n-k} && \text{(multiply and divide by } n!\text{)} \\ &= \frac{e^{-(\lambda+\mu)}}{n!} \sum_{k=0}^{n} \binom{n}{k}\lambda^k \mu^{n-k} \\ &= \frac{e^{-(\lambda+\mu)}\,(\lambda+\mu)^n}{n!}. && \text{(binomial theorem)} \end{aligned}

This is exactly the $\Pois(\lambda+\mu)$ PMF evaluated at $n$ . The mean $\lambda + \mu$ also follows directly from linearity of expectation.

Corollary 4.4 (Many independent Poissons)

If $X_1, \ldots, X_m$ are independent with $X_i \sim \Pois(\lambda_i)$ , then $\sum_{i=1}^m X_i \sim \Pois\bigl(\sum_{i=1}^m \lambda_i\bigr)$ . (Apply the theorem repeatedly; $X_1 + \cdots + X_{k}$ is independent of $X_{k+1}$ .)

Networking picture: merging. Two independent packet streams feed a router through separate buffers. If the numbers of packets arriving in each buffer during an interval are $\Pois(\lambda)$ and $\Pois(\mu)$ , the number of packets in the merged output buffer is $\Pois(\lambda + \mu)$ .

Splitting is the reverse operation: each arriving packet is routed to one of two outputs by an independent coin flip. Given the total count $X = n$ , the number $Y$ sent to the first output is the number of successes in $n$ independent Bernoulli $(p)$ trials. This is a “binomial with a random number of trials.”

Theorem 4.11 (Poisson splitting)

Let $X \sim \Pois(\lambda)$ , and suppose that, conditioned on $X = n$ , the random variable $Y$ satisfies

Y \mid X = n \;\sim\; \Bin(n, p).

Let $Z = X - Y$ be the remainder. Then

Y \sim \Pois(\lambda p), \qquad Z \sim \Pois\bigl(\lambda(1-p)\bigr), \qquad\text{and}\qquad Y \perp Z.

Proof (supplementary).

We compute the joint PMF of $(Y, Z)$ and show that it factors. Fix integers $a, b \ge 0$ . The event $\{Y = a, Z = b\}$ is the same as $\{X = a + b,\, Y = a\}$ (the total is $a + b$ , of which $a$ went to the first output). By the chain rule,

\begin{aligned} \PP(Y = a,\, Z = b) &= \PP(X = a + b)\,\PP(Y = a \mid X = a + b) \\ &= \frac{e^{-\lambda}\lambda^{a+b}}{(a+b)!} \cdot \binom{a+b}{a} p^a (1-p)^b && \text{(Poisson, then binomial)} \\ &= \frac{e^{-\lambda}\lambda^{a+b}}{(a+b)!} \cdot \frac{(a+b)!}{a!\,b!}\, p^a (1-p)^b && \text{(expand the binomial coefficient)} \\ &= e^{-\lambda}\,\frac{\lambda^{a}p^a}{a!}\cdot\frac{\lambda^{b}(1-p)^b}{b!} && \text{(cancel } (a+b)!\text{, split } \lambda^{a+b}\text{)} \\ &= \underbrace{\frac{e^{-\lambda p}(\lambda p)^a}{a!}}_{\Pois(\lambda p)\text{ PMF at } a} \cdot \underbrace{\frac{e^{-\lambda(1-p)}(\lambda(1-p))^b}{b!}}_{\Pois(\lambda(1-p))\text{ PMF at } b}, && \text{(split } e^{-\lambda} = e^{-\lambda p}e^{-\lambda(1-p)}\text{)} \end{aligned}

Marginal of $Y$ . Sum over $b \ge 0$ . The second factor is a PMF in $b$ , so it sums to $1$ , leaving $\PP(Y = a) = \frac{e^{-\lambda p}(\lambda p)^a}{a!}$ . Thus $Y \sim \Pois(\lambda p)$ . Summing over $a$ instead shows $Z \sim \Pois(\lambda(1-p))$ .

Independence. The joint PMF equals the product of the two marginals for every $(a, b)$ , which is the definition of $Y \perp Z$ .

Why the independence is surprising. If the total $X$ were a fixed number $n$ , then $Y$ and $Z = n - Y$ would be perfectly (negatively) dependent: learning $Y$ would tell you $Z$ exactly. It is the randomness of the total count that makes the two outputs independent. A large value of $Y$ suggests $X$ was large, which exactly offsets what it would otherwise say about $Z$ . This is special to the Poisson.

A Poisson(λ) buffer splits with probability p into Y ~ Poisson(λp) and with probability 1 − p into Z ~ Poisson(λ(1 − p))

Conditional PMFs and Conditional Expectation

Definition 4.4 (Conditional PMF)

Let $X, Y$ be discrete random variables and let $y$ satisfy $p_Y(y) > 0$ . The conditional PMF of $X$ given $Y = y$ is

p_{X\mid Y}(x \mid y) \triangleq \PP(X = x \mid Y = y) = \frac{p_{X,Y}(x, y)}{p_Y(y)}.

It is a genuine PMF in $x$ . For each fixed $y$ with $p_Y(y) > 0$ :

$p_{X\mid Y}(x \mid y) \ge 0$ , since it is a ratio of a probability and a positive number;
it sums to one over $x$ , by marginalizing the joint PMF: $\sum_x p_{X\mid Y}(x \mid y) = \frac{1}{p_Y(y)}\sum_x p_{X,Y}(x, y) = \frac{p_Y(y)}{p_Y(y)} = 1.$

So for each fixed value $y$ , “ $X$ given $Y = y$ ” is just another random variable with its own distribution, and we can take its expectation. (Note that the fraction is $p_{X,Y}(x,y)/p_Y(y)$ , a ratio of PMF values, rather than $\PP(X \cap Y)/\PP(Y)$ , since $X$ and $Y$ are random variables rather than events.)

Definition 4.5 (Conditional expectation given

Y = y

)

For $y$ with $p_Y(y) > 0$ , the conditional expectation of $X$ given $Y = y$ is

\EE[X \mid Y = y] \triangleq \sum_{x} x \cdot p_{X\mid Y}(x \mid y) = \sum_x x \cdot \PP(X = x \mid Y = y). \tag{4.74}

Like any expectation, $\EE[X \mid Y = y]$ is just a real number.

By “sweeping” through the values of $y$ in (4.74), we get a (possibly different) number $\EE[X \mid Y = y]$ for every value $y$ . That is, $y \mapsto \EE[X \mid Y = y]$ is a function of $y$ .

Definition 4.6 (Conditional expectation as a random variable)

Define $g(y) = \EE[X \mid Y = y]$ . The conditional expectation of $X$ given $Y$ is the random variable

\EE[X \mid Y] \triangleq g(Y).

It is a function of $Y$ , and therefore $\EE[X \mid Y]$ is itself a random variable: it takes the value $g(y)$ with probability $p_Y(y)$ .

Notation: $\EE[X \mid Y = y]$ versus $\EE[X \mid Y]$ . These are different objects. $\EE[X \mid Y = y]$ (lowercase $y$ , a specific value) is a number. $\EE[X \mid Y]$ (capital $Y$ ) is a random variable.

Example 4.3 (Die roll, conditioned on parity)

Roll a fair six-sided die. Let $X$ be the result and $Y = \mathbf{1}\{X \text{ is even}\}$ .

Given $Y = 1$ , $X$ is uniform on $\{2, 4, 6\}$ , so $\EE[X \mid Y = 1] = \frac{2 + 4 + 6}{3} = 4$ . Given $Y = 0$ , $X$ is uniform on $\{1, 3, 5\}$ , so $\EE[X \mid Y = 0] = \frac{1 + 3 + 5}{3} = 3$ . Hence

\EE[X \mid Y] = g(Y) = \begin{cases} 4 & \text{if } Y = 1, \\ 3 & \text{if } Y = 0, \end{cases} \qquad\text{i.e.}\qquad \EE[X \mid Y] = 3 + Y.

This is a random variable taking the values $3$ and $4$ , each with probability $\tfrac12$ . Notice that its mean is $\tfrac12(3) + \tfrac12(4) = 3.5 = \EE[X]$ . That is no coincidence, as the next lecture will show.

Law of Iterated Expectation (Tower Rule): TBD next lecture. This was on today’s agenda but was not reached in lecture. It states that $\EE\bigl[\EE[X \mid Y]\bigr] = \EE[X]$ ; the total expectation theorem of Section 4.5 is a special case.

Summary: Common Discrete Distributions

Distribution	PMF $\PP(X = k)$	Support	$\EE[X]$	$\Var(X)$
$\Bern(p)$	$p^k(1-p)^{1-k}$	$\{0, 1\}$	$p$	$p(1-p)$
$\Bin(n, p)$	$\binom{n}{k}p^k(1-p)^{n-k}$	$\{0, \ldots, n\}$	$np$	$np(1-p)$
$\Geom(p)$	$(1-p)^{k-1}p$	$\{1, 2, \ldots\}$	$\dfrac{1}{p}$	$\dfrac{1-p}{p^2}$
$\Pois(\lambda)$	$\dfrac{e^{-\lambda}\lambda^k}{k!}$	$\{0, 1, 2, \ldots\}$	$\lambda$	$\lambda$

Lecture 5: Iterated Expectation, Conditional Variance, and Continuous Random Variables

EECS 126 (UC Berkeley) · Spring 2026 · February 3, 2026

Reading: Bertsekas & Tsitsiklis, §4.2–4.3; Ch. 3.

These are my personal notes from Professor Kannan Ramchandran’s EECS 126 (Probability and Random Processes) course at UC Berkeley.

We open with a famous game whose expected payout is infinite. We then finish the story of conditional expectation from Lecture 4: the law of iterated expectation (the tower rule), which we use to analyze a random walk, and its companion for variances, the law of total variance. The second half of the lecture moves from discrete to continuous random variables, introducing probability density functions (PDFs) and cumulative distribution functions (CDFs).

The St. Petersburg Paradox

Example 5.1 (St. Petersburg game)

I toss a fair coin until I get a heads. If I toss $n$ times, I give you $\text{\textdollar}2^n$ . For example,

H \;\mapsto\; \$2, \qquad TH \;\mapsto\; \$4, \qquad TTH \;\mapsto\; \$8, \qquad \ldots

Question: how much would you pay to play this game?

Let $N$ be the number of tosses in the game and $W = 2^N$ the amount of money you win. The game ends on the first heads, so $N \sim \Geom(\tfrac12)$ :

\PP(N = n) = \underbrace{\tfrac12 \cdot \tfrac12 \cdots \tfrac12}_{n \text{ flips}} = 2^{-n}, \qquad n = 1, 2, 3, \ldots

(The first $n - 1$ flips are tails and the $n$ -th is heads; each has probability $\tfrac12$ .) By LOTUS,

\EE[W] = \sum_{n=1}^{\infty} 2^n \cdot \PP(N = n) = \sum_{n=1}^{\infty} 2^n \cdot \frac{1}{2^n} = \sum_{n=1}^{\infty} 1 = \infty.

The expected payout is infinite. This is the St. Petersburg paradox.

Why is it a paradox? If expected value were the whole story, you should be willing to pay any finite price to play. Yet almost nobody would pay more than a few dollars: half the time the game pays $\text{\textdollar}2$ , and $\PP(W \le \text{\textdollar}8) = \tfrac78$ . The infinite mean is driven entirely by astronomically rare, astronomically large payouts (each outcome $n$ contributes exactly $\text{\textdollar}1$ to the mean). Two standard resolutions:

Diminishing utility. Daniel Bernoulli argued that the value of money grows more slowly than the amount, e.g. logarithmically. The expected log-payout is finite: $\EE[\log_2 W] = \EE[N] = 2$ (the geometric mean from Lecture 4), which corresponds to a “fair price” of about $2^2 = \text{\textdollar}4$ .
Finite bankroll. No casino can pay more than some $\text{\textdollar}2^M$ . If payouts are capped at $2^M$ , then $\EE[W] = \sum_{n=1}^{M} 2^n 2^{-n} + \sum_{n=M+1}^{\infty} 2^M 2^{-n} = M + 2^M \cdot 2^{-M} = M + 1.$ Even a billion-dollar bankroll ( $2^{30} \approx 1.07\times 10^9$ ) gives an expected payout of only $\text{\textdollar}31$ .

Conditional Expectation as a Random Variable

Recall from Lecture 4 that for each value $y$ with $\PP(Y = y) > 0$ ,

\EE[X \mid Y = y] = \sum_x x \cdot \PP(X = x \mid Y = y)

is a number. Letting $g(y) = \EE[X \mid Y = y]$ , the conditional expectation $\EE[X \mid Y] \triangleq g(Y)$ is a function of $Y$ , and therefore $\EE[X \mid Y]$ is a random variable.

Example 5.2 (Tossing a fair coin a random number of times)

We toss a fair coin $N$ times, where $N$ is itself a random variable taking values in $\{1, 2, 3, \ldots\}$ . Let $X$ be the total number of heads. Given $N = n$ , the $n$ tosses are independent fair Bernoulli trials, so $X \mid N = n \sim \Bin(n, \tfrac12)$ .

$N = 1$ : one toss gives $0$ or $1$ heads, each with probability $\tfrac12$ :

\EE[X \mid N = 1] = 0 \cdot \tfrac12 + 1 \cdot \tfrac12 = \tfrac12 \qquad \text{(a number).}

$N = 2$ : two tosses give $0$ , $1$ , or $2$ heads, with probabilities $\tfrac14$ ( $TT$ ), $\tfrac12$ ( $HT$ or $TH$ ), and $\tfrac14$ ( $HH$ ):

\EE[X \mid N = 2] = 0 \cdot \tfrac14 + 1 \cdot \tfrac12 + 2 \cdot \tfrac14 = 1 \qquad \text{(a number).}

General $N = n$ : by the binomial mean from Lecture 4,

\EE[X \mid N = n] = n \cdot \tfrac12 = \frac{n}{2} \qquad \text{(a number).}

Sweeping over all values $n$ and replacing $n$ by the random variable $N$ ,

\underbrace{\EE[X \mid N]}_{\text{r.v.}} = \underbrace{\frac{N}{2}}_{\text{r.v.}}. \tag{5.9}

This is an equality of random variables: for every outcome of the experiment, the two sides take the same value.

The outcomes are not equally likely. For $N = 2$ , averaging the possible head counts as $\frac{0 + 1 + 2}{3} = 1$ happens to give the right answer, but only by coincidence: the three outcomes have probabilities $\tfrac14, \tfrac12, \tfrac14$ , not $\tfrac13$ each. For $N = 3$ the same shortcut would still give $\tfrac32$ by symmetry, but for a biased coin it fails immediately. Always weight by the conditional PMF.

The Law of Iterated Expectation (Tower Property)

Since $\EE[X \mid Y]$ is a random variable, we can take its expectation. The answer is remarkably simple.

Theorem 5.1 (Law of iterated expectation)

For random variables $X, Y$ with $\EE\bigl[|X|\bigr] < \infty$ ,

\EE\bigl[\EE[X \mid Y]\bigr] = \EE[X].

The inner expectation is over $X$ (with $Y$ held fixed); the outer expectation is over $Y$ .

Proof.

Write $\EE[X \mid Y] = g(Y)$ with $g(y) = \EE[X \mid Y = y]$ . The outer expectation is the expectation of a function of $Y$ , so by LOTUS (summing over the values $y$ with $\PP(Y = y) > 0$ ):

\begin{aligned} \EE\bigl[\EE[X \mid Y]\bigr] &= \sum_y \EE[X \mid Y = y] \cdot \PP(Y = y) && \text{(outer expectation, LOTUS)} \\ &= \sum_y \sum_x x \cdot \PP(X = x \mid Y = y) \cdot \PP(Y = y) && \text{(definition of } \EE[X \mid Y = y]) \\ &= \sum_y \sum_x x \cdot \PP(X = x,\, Y = y) && \text{(definition of conditional probability)} \\ &= \sum_x x \sum_y \PP(X = x,\, Y = y) && \text{(swap the order of summation)} \\ &= \sum_x x \cdot \PP(X = x) && \text{(marginalize out } Y) \\ &= \EE[X]. \end{aligned}

The third line uses $\PP(X = x \mid Y = y)\,\PP(Y = y) = \PP(X = x, Y = y)$ , i.e. $p_{X\mid Y}(x \mid y)\,p_Y(y) = p_{X,Y}(x, y)$ . Swapping the sums is justified by $\EE[|X|] < \infty$ (absolute convergence).

Intuition: an average of averages. $\EE[X \mid Y = y]$ is the average of $X$ within the “group” where $Y = y$ . The tower rule says: average these group averages, weighting each group by how likely it is, and you recover the overall average. The total expectation theorem from Lecture 4, $\EE[X] = \sum_i \PP(A_i)\,\EE[X \mid A_i]$ , is the special case where $Y$ records which event $A_i$ of a partition occurred.

A useful companion result lets us pull functions of $Y$ through the inner expectation: once we condition on $Y$ , any function of $Y$ is a known constant.

Theorem 5.2 (Taking out what is known)

For any function $f$ with $\EE\bigl[|f(Y)\,X|\bigr] < \infty$ ,

\EE\bigl[f(Y)\,X \mid Y\bigr] = f(Y)\,\EE[X \mid Y],

and consequently

\EE\bigl[f(Y)\,X\bigr] = \EE\bigl[f(Y)\,\EE[X \mid Y]\bigr]. \tag{5.13}

Proof.

Fix $y$ with $\PP(Y = y) > 0$ . Given $Y = y$ , the random variable $f(Y)$ equals the constant $f(y)$ , so it factors out of the conditional expectation:

\begin{aligned} \EE\bigl[f(Y)\,X \mid Y = y\bigr] &= \sum_x f(y)\,x\,\PP(X = x \mid Y = y) \\ &= f(y) \sum_x x\,\PP(X = x \mid Y = y) = f(y)\,\EE[X \mid Y = y]. \end{aligned}

Replacing $y$ by $Y$ gives $\EE[f(Y)X \mid Y] = f(Y)\,\EE[X \mid Y]$ . Now apply the tower rule to the random variable $f(Y)X$ :

\EE\bigl[f(Y)\,X\bigr] = \EE\bigl[\EE[f(Y)\,X \mid Y]\bigr] = \EE\bigl[f(Y)\,\EE[X \mid Y]\bigr].

Where the brackets go. In (5.13), $f(Y)$ stays inside the outer expectation, multiplied by the random variable $\EE[X \mid Y]$ . It is not true in general that $\EE[f(Y)X] = \EE[f(Y)]\cdot\EE[X \mid Y]$ ; the right-hand side of that would not even be a number. For instance, with $f(Y) = Y^2$ : $\EE[Y^2 X] = \EE\bigl[Y^2\,\EE[X \mid Y]\bigr]$ .

Example 5.3 (Random number of tosses, with

N \sim \Geom(p)

)

Suppose in Example 5.2 that $N$ is generated by a first coin with heads probability $p$ , tossed until its first heads, so that $N \sim \Geom(p)$ . We then toss a second, fair coin $N$ times and let $X$ be the number of heads. From (5.9), $\EE[X \mid N] = \frac{N}{2} = \frac12 \cdot N$ .

Mean. By the tower rule, linearity, and $\EE[N] = \frac1p$ for $N \sim \Geom(p)$ (Lecture 4),

\EE[X] = \EE\bigl[\EE[X \mid N]\bigr] = \EE\Bigl[\frac{N}{2}\Bigr] = \frac12\,\EE[N] = \frac12 \cdot \frac1p = \frac{1}{2p}.

Sanity check: if $p = 1$ , then $N = 1$ always, and one fair toss gives $\EE[X] = \frac12$ . ✓

A mixed moment. Taking out what is known with $f(N) = N$ , and using $\EE[N^2] = \frac{2-p}{p^2}$ from Lecture 4,

\EE[NX] = \EE\bigl[N\,\EE[X \mid N]\bigr] = \EE\Bigl[N \cdot \frac{N}{2}\Bigr] = \frac12\,\EE[N^2] = \frac{2 - p}{2p^2}.

The Symmetric Random Walk

Example 5.4 (Random walk on the integers)

A drunk man is walking along the integer line. He starts at the origin, $X_0 = 0$ . At each step, independently of everything before, he moves one unit right with probability $\tfrac12$ or one unit left with probability $\tfrac12$ . Let $X_n$ be his location after $n$ steps.

Number line with the walker at n and arrows to n − 1 and n + 1, each labeled probability 1/2

Questions: What is $\EE[X_n]$ ? What is $\EE[X_n^2]$ , and how far from the origin is he typically?

Write the $(n+1)$ -th step using two indicator random variables,

\mathbf{1}_{+}^{(n+1)} = \begin{cases} 1 & \text{if step } n+1 \text{ is } +1, \\ 0 & \text{otherwise,} \end{cases} \qquad \mathbf{1}_{-}^{(n+1)} = \begin{cases} 1 & \text{if step } n+1 \text{ is } -1, \\ 0 & \text{otherwise.} \end{cases}

Exactly one of them equals $1$ , so the step is $\mathbf{1}_{+}^{(n+1)} - \mathbf{1}_{-}^{(n+1)} \in \{+1, -1\}$ , and

X_{n+1} = X_n + \mathbf{1}_{+}^{(n+1)} - \mathbf{1}_{-}^{(n+1)}. \tag{5.19}

Notation. An indicator only takes the values $0$ and $1$ , so a step of $-1$ is written by subtracting the indicator $\mathbf{1}_{-}^{(n+1)}$ . The superscript $(n+1)$ records that each step is a fresh, independent coin flip. Similarly, the location $X_n$ and the step taken at time $n$ are different random variables, so we keep separate names for them.

The mean: 𝔼[Xₙ] = 0 for all n

Take expectations in (5.19). By linearity, and since $\EE[\mathbf{1}_{\pm}^{(n+1)}] = \PP(\text{step is } \pm 1) = \tfrac12$ ,

\EE[X_{n+1}] = \EE[X_n] + \EE\bigl[\mathbf{1}_{+}^{(n+1)}\bigr] - \EE\bigl[\mathbf{1}_{-}^{(n+1)}\bigr] = \EE[X_n] + \tfrac12 - \tfrac12 = \EE[X_n]. \tag{5.20}

So one step never changes the expected location.

Theorem 5.3 (Mean of the symmetric random walk)

For every $n \ge 0$ , $\EE[X_n] = 0$ .

Proof.

By induction on $n$ . Base case: $X_0 = 0$ is a constant, so $\EE[X_0] = 0$ . Inductive step: suppose $\EE[X_n] = 0$ . By (5.20), $\EE[X_{n+1}] = \EE[X_n] = 0$ . Unrolled, this is just a chain of equalities:

\EE[X_n] = \EE[X_{n-1}] = \EE[X_{n-2}] = \cdots = \EE[X_1] = \EE[X_0] = 0.

Two more ways to see it.

Sum of steps. Let $\xi_i \in \{+1, -1\}$ be the $i$ -th step, so $X_n = \xi_1 + \xi_2 + \cdots + \xi_n$ . Each step has mean $\EE[\xi_i] = (+1)\cdot\tfrac12 + (-1)\cdot\tfrac12 = 0$ , and by linearity $\EE[X_n] = \sum_{i=1}^n \EE[\xi_i] = 0$ .
Symmetry. Flipping the direction of every step turns the walk $X_n$ into $-X_n$ , and since left and right are equally likely, the flipped walk has exactly the same distribution. So $\EE[X_n] = \EE[-X_n] = -\EE[X_n]$ , which forces $\EE[X_n] = 0$ .

A zero mean does not mean the walker stays near the origin. He drifts away, but is equally likely to drift left or right, so the drifts cancel in the average. To measure how far he wanders, we need the second moment.

The second moment: 𝔼[Xₙ²] = n

Suppose $X_n = k$ . The next step is $\pm 1$ with probability $\tfrac12$ each, independent of the past, so

\PP\bigl(X_{n+1}^2 = (k+1)^2 \mid X_n = k\bigr) = \tfrac12, \qquad \PP\bigl(X_{n+1}^2 = (k-1)^2 \mid X_n = k\bigr) = \tfrac12.

These are the same two equally likely moves ( $k \to k+1$ and $k \to k-1$ ), just with the resulting location squared. Therefore

\begin{aligned} \EE\bigl[X_{n+1}^2 \mid X_n = k\bigr] &= \tfrac12 (k+1)^2 + \tfrac12 (k-1)^2 \\ &= \tfrac12 (k^2 + 2k + 1) + \tfrac12 (k^2 - 2k + 1) \\ &= k^2 + 1. \end{aligned}

The edge case $k = 0$ . When $k = 0$ , $(k+1)^2 = (k-1)^2 = 1$ , so the two events above are the same event, which has probability $1$ (not $\tfrac12$ ). The computation of $\EE[X_{n+1}^2 \mid X_n = 0]$ is still correct, because we are really averaging over the two moves: $\tfrac12 \cdot 1 + \tfrac12 \cdot 1 = 1 = 0^2 + 1$ .

We found $g(k) = \EE[X_{n+1}^2 \mid X_n = k] = k^2 + 1$ . Replacing the value $k$ by the random variable $X_n$ (the “pattern matching” step, exactly as in $\EE[X \mid N] = N/2$ ):

\EE\bigl[X_{n+1}^2 \mid X_n\bigr] = X_n^2 + 1.

By the law of iterated expectation,

\begin{aligned} \EE\bigl[X_{n+1}^2\bigr] &= \EE\bigl[\EE[X_{n+1}^2 \mid X_n]\bigr] && \text{(law of iterated expectation)} \\ &= \EE\bigl[X_n^2 + 1\bigr] \\ &= 1 + \EE\bigl[X_n^2\bigr]. && \text{(linearity)} \end{aligned}

Each step adds exactly $1$ to the second moment, starting from $\EE[X_0^2] = 0$ :

\EE[X_0^2] = 0, \quad \EE[X_1^2] = 1, \quad \EE[X_2^2] = 2, \quad \ldots \quad\Longrightarrow\quad \EE[X_n^2] = n,

(formally, by induction exactly as for the mean). In particular $\EE[X_{n+1}^2] = 1 + n$ .

Theorem 5.4 (Spread of the symmetric random walk)

For every $n \ge 0$ ,

\Var(X_n) = \EE[X_n^2] - (\EE[X_n])^2 = n - 0^2 = n, \qquad\text{so}\qquad \sigma_{X_n} = \sqrt{n}.

Cross-check with Lecture 4. With $X_n = \xi_1 + \cdots + \xi_n$ and independent steps, each step has $\Var(\xi_i) = \EE[\xi_i^2] - 0^2 = 1$ (since $\xi_i^2 = 1$ always). The variance of a sum of independent random variables is the sum of the variances, so $\Var(X_n) = n$ . Same answer, no conditioning needed.

Key takeaway. The walker’s expected position is $0$ forever, but his typical distance from the origin is $\sqrt{n}$ : after $10{,}000$ steps he is typically about $100$ units away. This $\sqrt{n}$ growth is the signature of diffusion, and it reappears throughout the course.

Conditional Variance and the Law of Total Variance

Definition 5.1 (Conditional variance)

For $y$ with $\PP(Y = y) > 0$ , the conditional variance of $X$ given $Y = y$ is the variance of the conditional distribution of $X$ :

\Var(X \mid Y = y) \triangleq \EE\Bigl[\bigl(X - \EE[X \mid Y = y]\bigr)^2 \Bigm| Y = y\Bigr] = \EE[X^2 \mid Y = y] - \bigl(\EE[X \mid Y = y]\bigr)^2.

This is a number for each $y$ , so $\Var(X \mid Y)$ , obtained by replacing $y$ with $Y$ , is a function of $Y$ , and hence a random variable.

(The second form follows exactly as in Lecture 3, applied to the conditional distribution of $X$ given $Y = y$ .)

Theorem 5.5 (Law of total variance)

For random variables $X, Y$ with $\EE[X^2] < \infty$ ,

\Var(X) = \EE\bigl[\Var(X \mid Y)\bigr] + \Var\bigl(\EE[X \mid Y]\bigr).

Proof.

Abbreviate $\mu(Y) = \EE[X \mid Y]$ , so that $\Var(X \mid Y) = \EE[X^2 \mid Y] - \mu(Y)^2$ .

Take expectations and use the tower rule on $\EE[X^2 \mid Y]$ :

\EE\bigl[\Var(X \mid Y)\bigr] = \EE\bigl[\EE[X^2 \mid Y]\bigr] - \EE\bigl[\mu(Y)^2\bigr] = \EE[X^2] - \EE\bigl[\mu(Y)^2\bigr].

Next, $\mu(Y)$ is a random variable; apply the variance formula to it, and use the tower rule on its mean, $\EE[\mu(Y)] = \EE[\EE[X\mid Y]] = \EE[X]$ :

\Var\bigl(\mu(Y)\bigr) = \EE\bigl[\mu(Y)^2\bigr] - \bigl(\EE[\mu(Y)]\bigr)^2 = \EE\bigl[\mu(Y)^2\bigr] - (\EE[X])^2.

Adding the two, the $\EE[\mu(Y)^2]$ terms cancel:

\begin{aligned} \EE\bigl[\Var(X \mid Y)\bigr] + \Var\bigl(\EE[X \mid Y]\bigr) &= \EE[X^2] - \EE\bigl[\mu(Y)^2\bigr] + \EE\bigl[\mu(Y)^2\bigr] - (\EE[X])^2 \\ &= \Var(X). \end{aligned}

Interpretation: within-group plus between-group. Group the outcomes by the value of $Y$ . Then $\EE[\Var(X \mid Y)]$ is the average spread within each group, and $\Var(\EE[X \mid Y])$ is the spread between the group means. Total spread is the sum of the two. Since both terms are nonnegative, $\Var(X) \ge \EE[\Var(X \mid Y)]$ : on average, learning $Y$ can only reduce our uncertainty about $X$ .

Example 5.5 (Variance of heads in a random number of tosses)

Continue the example with $N \sim \Geom(p)$ and $X \mid N \sim \Bin(N, \tfrac12)$ . From the binomial formulas of Lecture 4,

\EE[X \mid N] = \frac{N}{2}, \qquad \Var(X \mid N) = N \cdot \tfrac12 \cdot \tfrac12 = \frac{N}{4}.

Using $\EE[N] = \frac1p$ and $\Var(N) = \frac{1-p}{p^2}$ :

\begin{aligned} \EE\bigl[\Var(X \mid N)\bigr] &= \EE\Bigl[\frac{N}{4}\Bigr] = \frac14 \cdot \frac1p = \frac{1}{4p}, \\ \Var\bigl(\EE[X \mid N]\bigr) &= \Var\Bigl(\frac{N}{2}\Bigr) = \frac14\,\Var(N) = \frac{1-p}{4p^2}. \end{aligned}

By the law of total variance,

\Var(X) = \frac{1}{4p} + \frac{1-p}{4p^2} = \frac{p}{4p^2} + \frac{1-p}{4p^2} = \frac{1}{4p^2}.

Sanity check: if $p = 1$ , then $N = 1$ and $X \sim \Bern(\tfrac12)$ , whose variance is $\tfrac12\cdot\tfrac12 = \tfrac14$ . ✓

Continuous Random Variables

Continuous random variables (CRVs) let us measure uncertainty in a principled way for quantities that vary continuously, such as velocity, time, and distance.

Coming up. Important continuous distributions include the uniform, exponential, and gamma distributions, which we develop in upcoming lectures. The exponential is the continuous analog of the geometric (Lecture 4).

Why discrete tools break. Let $X \sim \operatorname{Uniform}[0,1]$ , a number chosen “uniformly at random” from $[0,1]$ . Then:

$\PP(X = x) = 0$ for any specific number $x$ (there are uncountably many equally likely candidates);
yet $\PP(X < \tfrac12) = \tfrac12$ .

So we cannot build probabilities by adding up point masses as we did with PMFs. Instead, we assign probabilities directly to sets that are “allowable events.”

Definition 5.2 (Allowable events (Borel sets))

The allowable events for a real-valued random variable include all intervals,

(-\infty, a), \qquad [a, b], \qquad (a, b], \qquad \ldots

together with everything obtainable from intervals by countable unions, countable intersections, and complements, e.g. $(a, b) \cup (c, d)$ . These are the Borel sets; they form the Borel $\sigma$ -algebra from Lecture 1. In practice, any “reasonable” set is allowable.

Recall that for a discrete random variable (DRV) and a set of values $B$ ,

\PP(X \in B) = \sum_{x \in B} p_X(x).

For a CRV, the sum over points becomes an integral of a density.

Definition 5.3 (Continuous random variable and PDF)

A random variable $X$ is continuous if there is a function $f_X : \RR \to [0, \infty)$ , called its probability density function (PDF), such that for every allowable set $B$ ,

\PP(X \in B) = \int_B f_X(x)\,dx.

A density must satisfy

f_X(x) \ge 0 \;\text{ for all } x, \qquad\text{and}\qquad \int_{-\infty}^{\infty} f_X(x)\,dx = 1.

In particular, $\PP(a \le X \le b) = \int_a^b f_X(x)\,dx$ is the area under the density between $a$ and $b$ .

A density curve f_X(x) with the area between a and b shaded, labeled P(a ≤ X ≤ b)

Theorem 5.6 (Single points have probability zero)

If $X$ is continuous, then $\PP(X = x) = 0$ for every $x \in \RR$ .

Proof.

Fix $x$ and any $\delta > 0$ . The event $\{X = x\}$ is contained in $\{x \le X < x + \delta\}$ , so

\PP(X = x) \le \PP(x \le X < x + \delta) = \int_x^{x+\delta} f_X(u)\,du.

(We use the dummy variable $u$ inside the integral so as not to confuse it with $x$ , which now appears in the limits.) As $\delta \to 0$ , this is the integral of $f_X$ over a shrinking interval, which tends to $0$ . If $f_X$ is continuous at $x$ , we can see this explicitly: the sliver is approximately a rectangle of height $f_X(x)$ and width $\delta$ , so the integral is $\approx f_X(x)\,\delta \to 0$ . Since $\PP(X = x)$ is smaller than a quantity that can be made arbitrarily small, $\PP(X = x) = 0$ .

A thin sliver under the density f_X(u) between x and x + δ, with area approximately f_X(x) δ

A density is not a probability. $f_X(x)$ has units of probability per unit length. Wherever $f_X$ is continuous,

f_X(x) = \lim_{\delta \to 0} \frac{\PP(x \le X < x + \delta)}{\delta}, \quad\text{i.e.}\quad \PP(x \le X < x + \delta) \approx f_X(x)\,\delta \text{ for small } \delta.

Don’t confuse $f_X(x)$ with a probability: it is a density. Only areas under $f_X$ are probabilities, so $f_X(x)$ itself may exceed $1$ , as the next example shows.

Example 5.6 (An unbounded density)

Let

f_X(x) = \begin{cases} \dfrac{1}{2\sqrt{x}} & 0 < x \le 1, \\[4pt] 0 & \text{otherwise.} \end{cases}

It is a valid PDF. Clearly $f_X \ge 0$ . For the total area, write $\frac{1}{2\sqrt{x}} = \frac12 x^{-1/2}$ and integrate:

\int_{-\infty}^{\infty} f_X(x)\,dx = \int_0^1 \tfrac12\, x^{-1/2}\,dx = \tfrac12 \Bigl[\, 2x^{1/2} \Bigr]_0^1 = \Bigl[\sqrt{x}\,\Bigr]_0^1 = 1 - 0 = 1.

It exceeds $1$ , and is even unbounded. $f_X(1) = \tfrac12$ , but $f_X(0.01) = \frac{1}{2(0.1)} = 5$ , and $f_X(x) \to \infty$ as $x \to 0^+$ . Nevertheless every probability is finite, e.g. $\PP(X \le 0.01) = \int_0^{0.01}\tfrac12 x^{-1/2}\,dx = \sqrt{0.01} = 0.1$ .

Graph of f_X(x) = 1/(2√x) on (0, 1], rising without bound as x approaches 0, with the area underneath shaded

The Cumulative Distribution Function

Definition 5.4 (Cumulative distribution function (CDF))

The cumulative distribution function of any random variable $X$ (discrete or continuous) is

F_X(x) \triangleq \PP(X \le x), \qquad x \in \RR.

For a continuous random variable,

F_X(x) = \int_{-\infty}^{x} f_X(u)\,du.

Theorem 5.7 (Properties of the CDF)

Every CDF satisfies:

Nondecreasing: if $x \le y$ then $F_X(x) \le F_X(y)$ , since $\{X \le x\} \subseteq \{X \le y\}$ .
Limits: $\displaystyle\lim_{x \to -\infty} F_X(x) = 0$ and $\displaystyle\lim_{x \to +\infty} F_X(x) = 1$ .
Intervals: $\PP(a < X \le b) = F_X(b) - F_X(a)$ , because $\{X \le b\}$ is the disjoint union of $\{X \le a\}$ and $\{a < X \le b\}$ .

Theorem 5.8 (Recovering the PMF or PDF from the CDF)

CRV: wherever $f_X$ is continuous, $f_X(x) = \frac{dF_X(x)}{dx}.$
Integer-valued DRV: for every integer $k$ , $\PP(X = k) = F_X(k) - F_X(k-1).$

Proof.

CRV: $F_X(x) = \int_{-\infty}^x f_X(u)\,du$ , so this is the fundamental theorem of calculus. DRV: because $X$ only takes integer values, there are no possible values strictly between $k-1$ and $k$ . So $\{X \le k\}$ is the disjoint union of $\{X \le k-1\}$ and $\{X = k\}$ , giving $F_X(k) = F_X(k-1) + \PP(X = k)$ . (For a discrete random variable taking non-integer values, use the jump of $F_X$ at the value instead.)

Picturing the two kinds of CDF. For an integer-valued DRV, $F_X$ is a staircase. On each open interval $(k-1, k)$ there are no possible values, so no new probability accumulates and $F_X$ stays flat, holding the value $F_X(k-1)$ . It does not drop back to zero: all the probability collected up through $k-1$ is still counted. At $k$ itself, $F_X$ jumps up by exactly $\PP(X = k)$ . For a CRV, single points carry no probability, so $F_X$ rises continuously with no jumps.

Left: the staircase CDF of an integer-valued discrete random variable, jumping by P(X = k) at k. Right: the continuous CDF of Uniform[0,1], rising linearly from 0 to 1

Corollary 5.1 (Consequences for continuous random variables)

If $X$ is continuous, then for all $a \le b$ :

\PP(a \le X \le b) = \int_a^b f_X(x)\,dx = F_X(b) - F_X(a),

\PP(X = a) = 0, \qquad \PP(X < a) = \PP(X \le a).

Because points do not contribute, it does not matter whether the endpoints of an interval are included: $\PP(a < X < b) = \PP(a \le X < b) = \PP(a < X \le b) = \PP(a \le X \le b)$ . This is false for discrete random variables, where $\PP(X < k)$ and $\PP(X \le k)$ differ by $\PP(X = k)$ .

Example: A Random Dart on a Disk

Example 5.7 (Random dart on the unit disk)

A dart lands uniformly at random on a disk of radius $1$ (the unit circle and its interior). “Uniformly” means the probability of landing in any region is proportional to its area:

\PP(\text{dart lands in region } R) = \frac{\text{Area}(R)}{\text{Area of full disk}} = \frac{\text{Area}(R)}{\pi (1)^2}.

Let $Y$ be the dart’s distance from the center. Find the PDF $f_Y(y)$ .

Unit disk with a shaded inner disk of radius y, labeled Y ≤ y, and a dart landing inside the unit disk

It is easiest to compute $F_Y(y) = \PP(Y \le y)$ first. For $0 \le y \le 1$ , the event $\{Y \le y\}$ means the dart lands in the shaded disk of radius $y$ :

F_Y(y) = \PP(Y \le y) = \frac{\text{Area shaded}}{\text{Area full disk}} = \frac{\pi y^2}{\pi (1)^2} = y^2.

Outside $[0,1]$ : $F_Y(y) = 0$ for $y < 0$ (distance is never negative) and $F_Y(y) = 1$ for $y > 1$ (the dart is always on the disk).

Differentiating,

f_Y(y) = \frac{dF_Y(y)}{dy} = \frac{d}{dy}\bigl(y^2\bigr) = 2y, \qquad 0 \le y \le 1,

and $f_Y(y) = 0$ otherwise. Check: $\int_0^1 2y\,dy = \bigl[y^2\bigr]_0^1 = 1$ . ✓

By the CDF,

\PP\bigl(Y \in (0.5, 0.6)\bigr) = F_Y(0.6) - F_Y(0.5) = 0.6^2 - 0.5^2 = 0.36 - 0.25 = 0.11.

Or, equivalently, by the PDF: $\int_{0.5}^{0.6} 2y\,dy = \bigl[y^2\bigr]_{0.5}^{0.6} = 0.11$ .

Three traps in this example.

$Y$ is not uniform on $[0,1]$ . The dart’s position is uniform on the disk, but its distance from the center is not: the density $2y$ grows with $y$ , because there is more area at larger radii. The ring $0.5 < Y < 0.6$ has probability $0.11$ , while the same-width band $0 < Y < 0.1$ near the center has only $0.1^2 = 0.01$ .
Don’t stop at the CDF. $y^2$ is $F_Y$ , not $f_Y$ ; the PDF requires the extra differentiation step.
$\frac{1}{\pi r^2}$ is a 2-D density. The constant $\frac{1}{\pi r^2}$ (here $\frac1\pi$ ) is the density of the dart’s position per unit area. It is not $f_Y$ , which is a density per unit length along the radius.

The CDF method. This “CDF first, then differentiate” strategy is the standard way to find the density of a function of a random variable.

Example: Meeting at Random Times

Not finished in lecture. We ran out of time after posing this problem. The full solution is given below.

Example 5.8 (Will A and B meet?)

$A$ and $B$ each arrive at a meeting spot at a uniformly random time between 4:00 pm and 5:00 pm, independently of each other. Whoever arrives first waits $10$ minutes and then leaves. What is $\PP(A \text{ and } B \text{ meet})$ ?

Setup. Measure time in minutes after 4:00 pm, so $A$ and $B$ are independent $\operatorname{Uniform}[0, 60]$ random variables. The pair $(A, B)$ is a point in the $60 \times 60$ square $[0,60]^2$ . Because $A$ and $B$ are independent and each uniform, the point $(A, B)$ is uniform on the square:

\PP\bigl((A, B) \in R\bigr) = \frac{\text{Area}(R)}{60 \times 60} = \frac{\text{Area}(R)}{3600}.

(Joint distributions of several continuous random variables are developed formally in upcoming lectures; this “probability $=$ area fraction” rule is the two-dimensional version of the uniform distribution, exactly as in the dart example.)

When do they meet? Whoever arrives first waits $10$ minutes, so they meet exactly when their arrival times differ by at most $10$ minutes:

\{A \text{ and } B \text{ meet}\} = \bigl\{|A - B| \le 10\bigr\}.

This is the shaded diagonal band in the figure below.

The 60 by 60 square of arrival times (A, B), with the diagonal band |A − B| ≤ 10 shaded as the meeting region and two corner triangles where they miss each other

Compute the complement. The two blue corner regions, where they miss each other, are right triangles:

$\{A - B > 10\}$ : $A$ arrives more than $10$ minutes after $B$ , so $B$ has already left. The triangle has vertices $(10,0), (60,0), (60,50)$ and legs of length $50$ .
$\{B - A > 10\}$ : the mirror image, with vertices $(0,10), (0,60), (50,60)$ , also with legs of length $50$ .

Each triangle has area $\tfrac12 \cdot 50 \cdot 50 = 1250$ , so together they cover $2500$ out of $3600$ :

\PP(\text{miss}) = \frac{2 \cdot 1250}{3600} = \frac{2500}{3600} = \frac{25}{36}.

Therefore

\PP(A \text{ and } B \text{ meet}) = 1 - \frac{25}{36} = \frac{11}{36} \approx 0.306.

Equivalently, the band has area $3600 - 2500 = 1100$ , and $\frac{1100}{3600} = \frac{11}{36}$ .

Checks and generalization.

Boundaries don’t matter. The lines $|A - B| = 10$ have zero area, so it makes no difference whether “waits $10$ minutes” includes the endpoint. This is the two-dimensional version of $\PP(X < a) = \PP(X \le a)$ .
General formula. With a window of length $T$ and waiting time $w \le T$ , the same picture gives $\PP(\text{meet}) = 1 - \Bigl(1 - \frac{w}{T}\Bigr)^2.$ With $w = 10$ , $T = 60$ : $1 - (5/6)^2 = 11/36$ . Sanity checks: $w = 0$ gives $0$ (they must arrive at the exact same instant), and $w = T$ gives $1$ .

Lecture 7: The Gaussian Distribution, Derived Distributions, and Order Statistics

EECS 126 (UC Berkeley) · Spring 2026 · February 10, 2026

Reading: Bertsekas & Tsitsiklis, §3.3, §4.1.

These are my personal notes from Professor Kannan Ramchandran’s EECS 126 (Probability and Random Processes) course at UC Berkeley.

Topics: the Gaussian distribution, change of variables (derived distributions), and order statistics.

The Gaussian Distribution

Definition 7.1 (Gaussian (normal) distribution)

$X \sim \mathcal{N}(\mu, \sigma^2)$ , with $\mu \in \RR$ and $\sigma > 0$ , if

f_X(x) = \frac{1}{\sqrt{2\pi}\,\sigma}\, e^{-(x-\mu)^2/2\sigma^2}, \qquad x \in \RR.

The standard normal is $Z \sim \mathcal{N}(0, 1)$ :

f_Z(z) = \frac{1}{\sqrt{2\pi}}\, e^{-z^2/2}, \qquad z \in \RR. \tag{7.2}

The constant $\frac{1}{\sqrt{2\pi}}$ is the normalization factor that makes $\int_{-\infty}^{\infty} f_Z(z)\,dz = 1$ .

Definition 7.2 (Standard normal CDF)

\Phi(z) \triangleq \PP(Z \le z) = \int_{-\infty}^{z} \frac{1}{\sqrt{2\pi}}\, e^{-u^2/2}\, du.

$u$ is a dummy variable of integration; $z$ appears only in the upper limit. $\Phi$ has no closed form, so its values are read from a table.

Theorem 7.1 (Key Gaussian integral)

For every $\alpha > 0$ ,

\int_{-\infty}^{\infty} e^{-\alpha x^2/2}\, dx = \sqrt{\frac{2\pi}{\alpha}},

or equivalently

I(\alpha) \triangleq \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-\alpha x^2/2}\, dx = \frac{1}{\sqrt{\alpha}} = \alpha^{-1/2}. \tag{7.5}

Proof.

Substitute $u = \sqrt{\alpha}\,x$ , so $du = \sqrt{\alpha}\,dx$ and $\alpha x^2 = u^2$ :

\int_{-\infty}^{\infty} e^{-\alpha x^2/2}\, dx = \int_{-\infty}^{\infty} e^{-u^2/2}\, \frac{du}{\sqrt{\alpha}} = \frac{1}{\sqrt{\alpha}} \int_{-\infty}^{\infty} e^{-u^2/2}\, du = \frac{1}{\sqrt{\alpha}} \cdot \sqrt{2\pi},

where the last step uses $\int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}} e^{-u^2/2}\,du = 1$ from (7.2). Dividing by $\sqrt{2\pi}$ gives (7.5).

Mean and Variance of the Standard Normal

Theorem 7.2 (Standard normal moments)

If $Z \sim \mathcal{N}(0,1)$ , then $\EE[Z] = 0$ and $\Var(Z) = 1$ .

Proof of 𝔼[Z] = 0.

\EE[Z] = \int_{-\infty}^{\infty} z\, f_Z(z)\, dz = \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, z\, e^{-z^2/2}\, dz.

The integrand $g(z) = z\,e^{-z^2/2}$ is odd, $g(-z) = -g(z)$ , so the two halves cancel:

\begin{aligned} \int_{0}^{\infty} z\, e^{-z^2/2}\, dz &= \Bigl[-e^{-z^2/2}\Bigr]_0^{\infty} = 0 - (-1) = 1, \\ \int_{-\infty}^{0} z\, e^{-z^2/2}\, dz &= \Bigl[-e^{-z^2/2}\Bigr]_{-\infty}^{0} = -1 - 0 = -1, \end{aligned}

and therefore $\EE[Z] = \frac{1}{\sqrt{2\pi}}(1 - 1) = 0$ .

Proof of Var(Z) = 1.

Differentiate $e^{-\alpha x^2/2}$ with respect to $\alpha$ :

\frac{d}{d\alpha}\Bigl(e^{-\alpha x^2/2}\Bigr) = -\frac{x^2}{2}\, e^{-\alpha x^2/2} \quad\Longrightarrow\quad x^2 e^{-\alpha x^2/2} = -2\, \frac{d}{d\alpha}\Bigl(e^{-\alpha x^2/2}\Bigr).

Then

\begin{aligned} \EE[Z^2] &= \int_{-\infty}^{\infty} x^2\, \frac{1}{\sqrt{2\pi}}\, e^{-x^2/2}\, dx \\ &= \left[\int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, x^2 e^{-\alpha x^2/2}\, dx\right]_{\alpha=1} \\ &= \left[\int_{-\infty}^{\infty} \frac{-2}{\sqrt{2\pi}}\, \frac{d}{d\alpha}\Bigl(e^{-\alpha x^2/2}\Bigr)\, dx\right]_{\alpha=1} \\ &= \left[-2\, \frac{d}{d\alpha}\left(\frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-\alpha x^2/2}\, dx\right)\right]_{\alpha=1} && \text{(Leibniz rule)} \\ &= \left[-2\, \frac{d}{d\alpha}\, I(\alpha)\right]_{\alpha=1} = \left[-2\, \frac{d}{d\alpha}\, \alpha^{-1/2}\right]_{\alpha=1} && \text{by (7.5)} \\ &= \left[-2 \left(-\tfrac12\, \alpha^{-3/2}\right)\right]_{\alpha=1} = \left[\alpha^{-3/2}\right]_{\alpha=1} = \left[\frac{1}{\sqrt{\alpha^3}}\right]_{\alpha=1} = 1. \end{aligned}

The Leibniz rule (differentiation under the integral sign) justifies moving $\frac{d}{d\alpha}$ outside the integral. Finally,

\Var(Z) = \EE[Z^2] - (\EE[Z])^2 = 1 - 0^2 = 1.

Why only $\alpha = 1$ ? I was confused why we set $\alpha = 1$ , and whether it could be something else. The identity $I(\alpha) = \alpha^{-1/2}$ holds for every $\alpha > 0$ , so $I$ is a function of $\alpha$ that we can differentiate, and differentiating in $\alpha$ is what produces the factor $x^2$ . We set $\alpha = 1$ only after differentiating, because the standard normal has $e^{-x^2/2}$ , i.e. $\alpha = 1$ . Setting $\alpha = 1$ first would leave the constant $I(1) = 1$ , whose derivative is $0$ .

Cross-check by integration by parts. With $u = x$ and $dv = \frac{1}{\sqrt{2\pi}}\, x e^{-x^2/2}\,dx$ , so $v = -\frac{1}{\sqrt{2\pi}}\, e^{-x^2/2}$ :

\EE[Z^2] = \left[-\frac{x}{\sqrt{2\pi}}\, e^{-x^2/2}\right]_{-\infty}^{\infty} + \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, e^{-x^2/2}\, dx = 0 + 1 = 1.

Linear Transformations of Gaussians

Theorem 7.3 (Shifting and scaling a standard normal)

If $Z \sim \mathcal{N}(0,1)$ , $\sigma > 0$ , and $X = \sigma Z + \mu$ , then $X \sim \mathcal{N}(\mu, \sigma^2)$ .

Proof.

Start with the CDF of $X$ . Since $\sigma > 0$ , dividing by $\sigma$ preserves the inequality:

F_X(x) = \PP(X \le x) = \PP(\sigma Z + \mu \le x) = \PP\left(Z \le \frac{x - \mu}{\sigma}\right) = F_Z(z), \qquad z = \frac{x - \mu}{\sigma}.

Differentiate with the chain rule, using $\frac{dz}{dx} = \frac{d}{dx}\left(\frac{x}{\sigma} - \frac{\mu}{\sigma}\right) = \frac{1}{\sigma}$ :

\begin{aligned} f_X(x) = \frac{d}{dx} F_X(x) = \frac{d}{dx} F_Z(z) = f_Z(z)\, \frac{dz}{dx} &= \frac{1}{\sqrt{2\pi}}\, e^{-z^2/2} \cdot \frac{1}{\sigma} \\ &= \frac{1}{\sqrt{2\pi}\,\sigma}\, e^{-(x-\mu)^2/2\sigma^2}. \end{aligned}

Is $f_Z(z) \cdot \frac{1}{\sigma}$ already the answer? I was unsure whether the line $f_X(x) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2} \cdot \frac{1}{\sigma}$ was correct. It is: substituting $z = \frac{x-\mu}{\sigma}$ gives exactly the $\mathcal{N}(\mu, \sigma^2)$ density. The factor $\frac{1}{\sigma}$ in front comes from $\frac{dz}{dx}$ .

Should $\mu$ and $\sigma$ have a subscript $Z$ ? I was also confused about this. No: $Z$ always has mean $0$ and variance $1$ . Here $\mu$ and $\sigma$ are the shift and scale applied to $Z$ , and they become the mean and standard deviation of $X$ : $\EE[X] = \sigma\,\EE[Z] + \mu = \mu$ and $\Var(X) = \sigma^2 \Var(Z) = \sigma^2$ . If anything, they are $\mu_X$ and $\sigma_X$ .

Corollary 7.1 (Standardization)

If $X \sim \mathcal{N}(\mu, \sigma^2)$ , then $Z = \dfrac{X - \mu}{\sigma} \sim \mathcal{N}(0, 1)$ , and

F_X(x) = \Phi\!\left(\frac{x - \mu}{\sigma}\right).

Symmetry of Φ and Standardization

Theorem 7.4 (Symmetry of the standard normal CDF)

For every $y \in \RR$ ,

\Phi(-y) = 1 - \Phi(y).

So a table only needs $\Phi(y)$ for $y \ge 0$ .

Proof.

$f_Z$ is even: $f_Z(-v) = f_Z(v)$ . Substitute $u = -v$ , $du = -dv$ :

\begin{aligned} \Phi(-y) = \int_{-\infty}^{-y} f_Z(u)\, du &= \int_{\infty}^{y} f_Z(-v)\,(-dv) = \int_{y}^{\infty} f_Z(v)\, dv \\ &= \PP(Z \ge y) = 1 - \PP(Z < y) = 1 - \Phi(y), \end{aligned}

using $\PP(Z < y) = \PP(Z \le y)$ for a continuous random variable.

Standard normal density with equal shaded tails: the left tail below −y is Φ(−y) and the right tail above y is 1 − Φ(y)

Example 7.1 (Annual rainfall)

The annual rainfall in a city is $X \sim \mathcal{N}(60, 20^2)$ , in inches. Find $\PP(X > 80)$ .

Standardize with $Z = \frac{X - 60}{20} \sim \mathcal{N}(0,1)$ :

\begin{aligned} \PP(X > 80) &= \PP\left(\frac{X - 60}{20} > \frac{80 - 60}{20}\right) \\ &= \PP\left(\frac{X - 60}{20} > 1\right) = \PP(Z > 1) \\ &= 1 - \PP(Z \le 1) = 1 - \Phi(1) = 1 - 0.8413 = 0.1587. \end{aligned}

Standard normal density with the right tail above z = 1 shaded, labeled P(Z > 1)

Why $1 - \Phi(1)$ ? I was confused why the answer is $1 - \Phi(1)$ rather than $\Phi(1)$ . $\Phi(1)$ is the area to the left of $1$ ; $\PP(Z > 1)$ is the area to the right, which is its complement.

Sums of Independent Gaussians

Theorem 7.5 (Sum of independent Gaussians)

If $X \sim \mathcal{N}(\mu_X, \sigma_X^2)$ and $Y \sim \mathcal{N}(\mu_Y, \sigma_Y^2)$ are independent, then

X + Y \sim \mathcal{N}\bigl(\mu_X + \mu_Y,\; \sigma_X^2 + \sigma_Y^2\bigr).

The parameters follow from Lecture 4: $\EE[X+Y] = \EE[X] + \EE[Y] = \mu_X + \mu_Y$ , and by independence $\Var(X+Y) = \Var(X) + \Var(Y) = \sigma_X^2 + \sigma_Y^2$ . The content of the theorem is that $X + Y$ is again Gaussian. This can be shown with a convolution, or in one line with moment generating functions.

Proof: MGF proof TBD in a later lecture.

Theorem 7.6 (Converse)

If $X$ and $Y$ are independent and $X + Y$ is Gaussian, then $X$ and $Y$ are each Gaussian (allowing a constant, i.e. variance $0$ ).

This direction is much harder; its proof is beyond the scope of this course.

Derived Distributions

Given $X$ with known density $f_X$ and $Y = g(X)$ , find $f_Y$ .

Theorem 7.7 (CDF method)

F_Y(y) = \PP(Y \le y) = \PP\bigl(g(X) \le y\bigr) = \PP\bigl(X \in \{x : g(x) \le y\}\bigr) = \int_{\{x \,:\, g(x) \le y\}} f_X(x)\, dx,

and then

f_Y(y) = \frac{dF_Y(y)}{dy}.

If $g$ is strictly increasing, $\{x : g(x) \le y\} = \{x \le g^{-1}(y)\}$ , so $F_Y(y) = F_X\bigl(g^{-1}(y)\bigr)$ . If $g$ is strictly decreasing, the inequality flips: $\{x : g(x) \le y\} = \{x \ge g^{-1}(y)\}$ , so $F_Y(y) = 1 - F_X\bigl(g^{-1}(y)\bigr)$ .

Corollary 7.2 (Monotone change of variables)

If $g$ is strictly monotone and differentiable, then

f_Y(y) = f_X\bigl(g^{-1}(y)\bigr) \left| \frac{d\, g^{-1}(y)}{dy} \right|. \tag{7.21}

Proof.

Increasing $g$ : $\frac{d}{dy} F_X(g^{-1}(y)) = f_X(g^{-1}(y)) \frac{d g^{-1}(y)}{dy}$ , with $\frac{d g^{-1}(y)}{dy} > 0$ . Decreasing $g$ : $\frac{d}{dy}\bigl[1 - F_X(g^{-1}(y))\bigr] = -f_X(g^{-1}(y)) \frac{d g^{-1}(y)}{dy}$ , with $\frac{d g^{-1}(y)}{dy} < 0$ . In both cases the result is (7.21).

For $X = \sigma Z + \mu$ : $g^{-1}(x) = \frac{x-\mu}{\sigma}$ and $\frac{d g^{-1}(x)}{dx} = \frac{1}{\sigma}$ , which is the factor $\frac{1}{\sigma}$ found in Section 7.3.

Example 7.2 (Square root of a uniform)

Let $X \sim \operatorname{Uniform}[0,1]$ and $Y = \sqrt{X}$ . For $0 \le y \le 1$ , since $\sqrt{x}$ is increasing on $[0,1]$ and both sides are nonnegative,

F_Y(y) = \PP(Y \le y) = \PP(\sqrt{X} \le y) = \PP(X \le y^2) = F_X(y^2) = \int_0^{y^2} 1\, dx = \Bigl[x\Bigr]_0^{y^2} = y^2 - 0 = y^2,

with $F_Y(y) = 0$ for $y < 0$ and $F_Y(y) = 1$ for $y > 1$ . Differentiating,

f_Y(y) = \frac{d}{dy} F_Y(y) = \frac{d}{dy}\bigl(y^2\bigr) = 2y, \qquad 0 \le y \le 1.

Check with (7.21): $g^{-1}(y) = y^2$ , $\frac{d g^{-1}(y)}{dy} = 2y$ , $f_X(y^2) = 1$ , so $f_Y(y) = 1 \cdot 2y = 2y$ .

$Y = \sqrt{X}$ has the same density $2y$ as the distance of a random dart from the center of the unit disk (Lecture 5).

Order Statistics

Let $X_1, X_2, \ldots, X_n$ be i.i.d. samples with common density $f_X$ and CDF $F_X$ .

Definition 7.3 (Order statistics)

X^{(k)} \triangleq \text{the } k\text{-th smallest of } X_1, X_2, \ldots, X_n, \qquad k = 1, \ldots, n.

In particular $X^{(1)} = \min_i X_i$ and $X^{(n)} = \max_i X_i$ .

Example 7.3 (Ordering three samples)

$n = 3$ , $X_i \sim \operatorname{Uniform}(0,1)$ , observed $(X_1, X_2, X_3) = (0.3,\, 0.8,\, 0.4)$ . Then

X^{(1)} = 0.3, \qquad X^{(2)} = 0.4, \qquad X^{(3)} = 0.8.

Question: what is $f_{X^{(k)}}(x)$ , the density of $X^{(k)}$ ?

By the definition of a density,

\PP\bigl(X^{(k)} \in (x, x + dx)\bigr) \approx f_{X^{(k)}}(x)\, dx.

For the $k$ -th smallest sample to lie in $(x, x+dx)$ , we need

$k - 1$ samples in $(-\infty, x)$ : probability $\bigl[F_X(x)\bigr]^{k-1}$ ;
$1$ sample in $(x, x + dx)$ : probability $f_X(x)\,dx$ ;
$n - k$ samples in $(x + dx, \infty)$ : probability $\bigl[1 - F_X(x)\bigr]^{n-k}$ ,

where the probabilities multiply by independence.

Number line with k − 1 samples left of x, the k-th order statistic in the interval (x, x + dx), and n − k samples to the right

Count the ways to assign the $n$ samples to the three regions:

which sample lies in $(x, x+dx)$ : $n$ ways;
which $k-1$ of the remaining $n-1$ lie to the left: $\binom{n-1}{k-1}$ ways;
the remaining $n-k$ lie to the right: $1$ way.

n \binom{n-1}{k-1} = n \cdot \frac{(n-1)!}{(k-1)!\,\bigl((n-1)-(k-1)\bigr)!} = \frac{n!}{(k-1)!\,(n-k)!}.

Why only $1$ way for the third group? I was confused whether the right-hand group contributes $\binom{\,\cdot\,}{n-k}$ more choices. It does not: once the middle sample and the $k-1$ left samples are chosen, the other $n-k$ samples are forced to the right. (Two or more samples landing in $(x, x+dx)$ has probability of order $(dx)^2$ , which is negligible.)

Theorem 7.8 (Density of the

k

-th order statistic)

f_{X^{(k)}}(x) = \frac{n!}{(k-1)!\,(n-k)!}\, \bigl[F_X(x)\bigr]^{k-1} \bigl[1 - F_X(x)\bigr]^{n-k} f_X(x). \tag{7.28}

Proof.

Multiply the number of assignments by the probability of each:

f_{X^{(k)}}(x)\, dx = \frac{n!}{(k-1)!\,(n-k)!}\, \bigl[F_X(x)\bigr]^{k-1} \bigl[1 - F_X(x)\bigr]^{n-k} f_X(x)\, dx,

and divide both sides by $dx$ .

Why $y$ ? I was confused why the formula was written in $y$ rather than $x$ . The argument is a dummy variable: $f_{X^{(k)}}(y)$ and $f_{X^{(k)}}(x)$ are the same function. What matters is that one variable is used throughout, and that $F_X$ and $f_X$ are the CDF and PDF of a single sample, not of $X^{(k)}$ .

Check, $k = n$ (the maximum). $\PP(X^{(n)} \le x) = \PP(\text{all } X_i \le x) = \bigl[F_X(x)\bigr]^n$ , and $\frac{d}{dx}\bigl[F_X(x)\bigr]^n = n\bigl[F_X(x)\bigr]^{n-1} f_X(x)$ , matching (7.28) with $k = n$ .

Corollary 7.3 (Uniform samples)

If $X_i \sim \operatorname{Uniform}[0,1]$ , then $f_X(x) = 1$ and $F_X(x) = x$ on $[0,1]$ , so

f_{X^{(k)}}(x) = \frac{n!}{(k-1)!\,(n-k)!}\, x^{k-1} (1 - x)^{n-k}, \qquad 0 \le x \le 1. \tag{7.30}

On [0, 1], the Uniform density f_X(x) = 1 is a flat line and the CDF F_X(x) = x is the diagonal

Example 7.4 (Third smallest of four uniforms)

$X_i \sim \operatorname{Uniform}[0,1]$ , $n = 4$ . Find $\PP\bigl(X^{(3)} < \tfrac12\bigr)$ .

With $n = 4$ , $k = 3$ : $\frac{n!}{(k-1)!\,(n-k)!} = \frac{4!}{2!\,1!} = \frac{24}{2} = 12$ . By (7.30),

\begin{aligned} \PP\bigl(X^{(3)} < \tfrac12\bigr) = F_{X^{(3)}}\bigl(\tfrac12\bigr) &= \int_0^{1/2} f_{X^{(3)}}(x)\, dx = \int_0^{1/2} 12\, x^2 (1 - x)\, dx \\ &= 12 \int_0^{1/2} \bigl(x^2 - x^3\bigr)\, dx = 12 \left[\frac{x^3}{3} - \frac{x^4}{4}\right]_0^{1/2} \\ &= 12 \left(\frac{1}{24} - \frac{1}{64}\right) = 12 \left(\frac{8}{192} - \frac{3}{192}\right) = 12 \cdot \frac{5}{192} = \frac{60}{192} = \frac{5}{16}. \end{aligned}

Check: $X^{(3)} < \tfrac12$ exactly when at least $3$ of the $4$ samples are below $\tfrac12$ . The number below $\tfrac12$ is $\Bin(4, \tfrac12)$ , so

\PP\bigl(X^{(3)} < \tfrac12\bigr) = \binom{4}{3}\Bigl(\tfrac12\Bigr)^4 + \binom{4}{4}\Bigl(\tfrac12\Bigr)^4 = \frac{4}{16} + \frac{1}{16} = \frac{5}{16}.

Example 7.5 (Expected minimum by symmetry)

$X_i \sim \operatorname{Uniform}[0,1]$ , $n = 10$ . Find $\EE[X^{(1)}]$ , the expected smallest sample.

The $10$ samples cut $[0,1]$ into $11$ gaps:

\Delta_1 = X^{(1)} - 0, \qquad \Delta_i = X^{(i)} - X^{(i-1)} \;\; (2 \le i \le 10), \qquad \Delta_{11} = 1 - X^{(10)}.

Ten points on [0, 1] labeled X^(1) through X^(10), cutting the interval into gaps Δ1 through Δ11

The sum telescopes:

\sum_{i=1}^{11} \Delta_i = (X^{(1)} - 0) + (X^{(2)} - X^{(1)}) + \cdots + (X^{(10)} - X^{(9)}) + (1 - X^{(10)}) = 1.

By symmetry, all $11$ gaps have the same distribution, so $\EE[\Delta_i] = \EE[\Delta_1]$ for every $i$ . Taking expectations,

\EE\Bigl[\sum_{i=1}^{11} \Delta_i\Bigr] = \sum_{i=1}^{11} \EE[\Delta_i] = 11\, \EE[\Delta_1] = \EE[1] = 1 \quad\Longrightarrow\quad \EE[X^{(1)}] = \EE[\Delta_1] = \frac{1}{11}.

Do the gaps really sum to $1$ ? I was unsure about this. They do: the gaps tile $[0,1]$ exactly, which is the telescoping sum above.

Why are the gaps symmetric? Place $11$ independent uniform points on a circle of circumference $1$ and cut the circle at one of them. The other $10$ points are i.i.d. uniform on the resulting interval $[0,1]$ , and the $11$ arcs between consecutive points are exactly the gaps $\Delta_1, \ldots, \Delta_{11}$ . On the circle no arc is special, so all gaps have the same distribution.

Cross-check with (7.30). With $k = 1$ : $f_{X^{(1)}}(x) = \frac{10!}{0!\,9!}(1-x)^9 = 10(1-x)^9$ . Substituting $u = 1 - x$ ,

\EE[X^{(1)}] = \int_0^1 x \cdot 10(1-x)^9\, dx = \int_0^1 (1 - u)\, 10\, u^9\, du = 10\left(\frac{1}{10} - \frac{1}{11}\right) = 10 \cdot \frac{1}{110} = \frac{1}{11}.

General case. $X^{(k)} = \Delta_1 + \cdots + \Delta_k$ , so for $n$ uniform samples,

\EE[X^{(k)}] = k\, \EE[\Delta_1] = \frac{k}{n+1}.

Next lecture: moment generating functions (MGFs).

Lecture 8: Sums of Independent Random Variables and Moment Generating Functions

EECS 126 (UC Berkeley) · Spring 2026 · February 12, 2026

Reading: Bertsekas & Tsitsiklis, §4.4–4.6.

These are my personal notes from Professor Kannan Ramchandran’s EECS 126 (Probability and Random Processes) course at UC Berkeley.

Topics: sums of independent random variables (convolution), tail bounds, and moment generating functions (MGFs).

Discrete Convolution

Example 8.1 (Sum of two independent discrete random variables)

Let $X$ and $Y$ be independent, each taking values in $\{0, 1, 2, 3\}$ , with

p_X = (0.1,\ 0.2,\ 0.3,\ 0.4), \qquad p_Y = (0.4,\ 0.3,\ 0.2,\ 0.1)

at the values $0, 1, 2, 3$ respectively. Let $Z = X + Y$ . Find $\PP(Z = 3)$ .

Bar charts of the PMFs: p_X = 0.1, 0.2, 0.3, 0.4 and p_Y = 0.4, 0.3, 0.2, 0.1 at the values 0, 1, 2, 3

$Z = 3$ occurs exactly for the pairs $(x, y) \in \{(0,3), (1,2), (2,1), (3,0)\}$ , i.e. $y = 3 - x$ :

\begin{aligned} \PP(Z = 3) = \PP(X + Y = 3) &= \sum_{(x,y)\,:\, x + y = 3} \PP(X = x,\, Y = y) \\ &= \sum_{(x,y)\,:\, x + y = 3} \PP(X = x)\, \PP(Y = y) \qquad \text{(independence)} \\ &= \sum_{x} \PP(X = x)\, \PP(Y = 3 - x) \\ &= p_X(0)\,p_Y(3) + p_X(1)\,p_Y(2) + p_X(2)\,p_Y(1) + p_X(3)\,p_Y(0) \\ &= (0.1)(0.1) + (0.2)(0.2) + (0.3)(0.3) + (0.4)(0.4) \\ &= 0.01 + 0.04 + 0.09 + 0.16 = 0.30. \end{aligned}

To see the products line up, flip $p_Y$ and shift it by $3$ : plot $p_Y(3 - x)$ against $x$ , directly under $p_X(x)$ , then multiply bar by bar and add.

p_X(x) above the flipped and shifted p_Y(3 − x); multiplying bar by bar gives 0.01 + 0.04 + 0.09 + 0.16 = 0.30 = P(Z = 3)

Definition 8.1 (Discrete convolution)

If $X$ and $Y$ are independent and integer-valued, then $Z = X + Y$ has PMF

p_Z(z) = (p_X * p_Y)(z) \triangleq \sum_{x} p_X(x)\, p_Y(z - x). \tag{8.3}

Here $x$ is a dummy summation variable.

Applying (8.3) at every $z$ in the example gives the full PMF of $Z$ :

Bar chart of the PMF of Z = X + Y: 0.04, 0.11, 0.20, 0.30, 0.20, 0.11, 0.04 at z = 0 through 6, with z = 3 highlighted

Check: $0.04 + 0.11 + 0.20 + 0.30 + 0.20 + 0.11 + 0.04 = 1$ .

Continuous Convolution

Theorem 8.1 (Density of a sum of independent random variables)

If $X$ and $Y$ are independent continuous random variables and $Z = X + Y$ , then

f_Z(z) = (f_X * f_Y)(z) \triangleq \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, dx. \tag{8.4}

Here $x$ is a dummy variable of integration.

Proof by conditioning.

Find the conditional CDF of $Z$ given $X = x$ , then differentiate:

\begin{aligned} F_{Z \mid X}(z \mid x) = \PP(Z \le z \mid X = x) &= \PP(X + Y \le z \mid X = x) \\ &= \PP(x + Y \le z \mid X = x) \\ &= \PP(Y \le z - x \mid X = x) \\ &= \PP(Y \le z - x) && \text{(independence)} \\ &= F_Y(z - x). \end{aligned}

Differentiating in $z$ ,

f_{Z \mid X}(z \mid x) = \frac{d}{dz} F_Y(z - x) = f_Y(z - x).

Multiply both sides by the marginal $f_X(x)$ ; the left side becomes the joint PDF of $X$ and $Z$ :

f_{X,Z}(x, z) = f_{Z \mid X}(z \mid x)\, f_X(x) = f_Y(z - x)\, f_X(x).

To get the marginal $f_Z$ , integrate over all possible $x$ :

f_Z(z) = \int_{-\infty}^{\infty} f_{X,Z}(x, z)\, dx = \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, dx.

Why does the conditioning drop? I was unsure why we can remove “ $\mid X = x$ ”. Once $X = x$ is fixed, the only random variable left in $\{Y \le z - x\}$ is $Y$ , and $Y$ is independent of $X$ , so conditioning on $X = x$ does not change its distribution.

Proof by a thin strip.

$\PP(z \le Z \le z + \delta)$ is the probability that $(X, Y)$ lands in the strip between the lines $y = z - x$ and $y = z + \delta - x$ :

\begin{aligned} \PP(z \le Z \le z + \delta) &= \int_{-\infty}^{\infty} \int_{z - x}^{z - x + \delta} f_{X,Y}(x, y)\, dy\, dx \\ &= \int_{-\infty}^{\infty} \int_{z - x}^{z - x + \delta} f_X(x)\, f_Y(y)\, dy\, dx && \text{(independence)} \\ &= \int_{-\infty}^{\infty} f_X(x) \left[\int_{z - x}^{z - x + \delta} f_Y(y)\, dy\right] dx \\ &\approx \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, \delta\, dx, \end{aligned}

where the inner integral over an interval of width $\delta$ is approximated by a rectangle of height $f_Y(z - x)$ . Since also $\PP(z \le Z \le z + \delta) \approx f_Z(z)\,\delta$ ,

f_Z(z)\, \delta \approx \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, \delta\, dx,

and dividing by $\delta$ and letting $\delta \to 0$ gives (8.4).

The thin strip between the lines x + y = z and x + y = z + δ in the (x, y) plane

The strip has negative slope because $Z = X + Y$ means $Y = Z - X$ . What we integrate over the strip is the joint density, so $\PP(z \le Z \le z+\delta)$ is the probability mass of the strip, not its area.

Example: Sum of Two Uniforms

Example 8.2 (Sum of two independent uniforms)

Let $X, Y \sim \operatorname{Uniform}[0,1]$ be independent, so $f_X(x) = \frac{1}{b - a} = \frac{1}{1 - 0} = 1$ on $[0,1]$ , and the same for $f_Y$ . Find $f_Z$ for $Z = X + Y$ .

By (8.4), writing each density as an indicator,

f_Z(z) = \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, dx = \int_{-\infty}^{\infty} \mathbf{1}\{0 \le x \le 1\} \cdot \mathbf{1}\{0 \le z - x \le 1\}\, dx.

The second indicator is $1$ when $z - 1 \le x \le z$ . So the integrand is $1$ exactly on $\bigl[\max(0, z - 1),\, \min(1, z)\bigr]$ , and $f_Z(z)$ is the length of that interval:

\begin{aligned} 0 \le z \le 1: &\quad f_Z(z) = \int_0^{z} 1\, dx = z, \\ 1 \le z \le 2: &\quad f_Z(z) = \int_{z-1}^{1} 1\, dx = 1 - (z - 1) = 2 - z, \end{aligned}

and $f_Z(z) = 0$ otherwise. Check: the triangle has area $\tfrac12 \cdot 2 \cdot 1 = 1$ .

Left: the box f_X(x) on [0, 1] overlapping the sliding box f_Y(z − x) on [z − 1, z]. Right: the triangular density f_Z(z) on [0, 2] with peak 1 at z = 1

Sliding overlap. As a function of $x$ , $f_Y(z - x)$ is a box on $[z - 1, z]$ that slides to the right as $z$ increases. $f_Z(z)$ is the length of its overlap with the box $[0,1]$ : it grows from $0$ at $z = 0$ , is largest at $z = 1$ (exact overlap), and shrinks back to $0$ at $z = 2$ . The same flip-and-slide picture is how convolution is used in signal processing.

Bounding Tail Probabilities

Example 8.3 (How big must a buffer be?)

Over $n$ time slots, $P_i$ packets arrive in slot $i$ , where $P_1, \ldots, P_n \sim \Pois(1)$ are i.i.d. A buffer holds $5n$ packets. How likely is it to overflow? We want to bound

\PP\Bigl(\sum_{i=1}^{n} P_i > 5n\Bigr).

Theorem 8.2 (Markov’s inequality)

If $X \ge 0$ and $a > 0$ , then

\PP(X \ge a) \le \frac{\EE[X]}{a}.

Proof.

Pointwise, $X \ge a\,\mathbf{1}\{X \ge a\}$ : if $X \ge a$ the right side is $a \le X$ , and otherwise it is $0 \le X$ . Taking expectations, $\EE[X] \ge a\,\EE\bigl[\mathbf{1}\{X \ge a\}\bigr] = a\,\PP(X \ge a)$ .

Let $S_n = \sum_{i=1}^n P_i \ge 0$ . By linearity of expectation and $\EE[P_i] = 1$ for $\Pois(1)$ ,

\PP(S_n > 5n) \le \PP(S_n \ge 5n) \le \frac{\EE[S_n]}{5n} = \frac{\sum_{i=1}^n \EE[P_i]}{5n} = \frac{\sum_{i=1}^n 1}{5n} = \frac{n}{5n} = \frac15.

Too loose. The bound $\frac15$ does not shrink as $n$ grows, even though $S_n/n$ stays close to its mean $1$ , far below $5$ , for large $n$ . Markov only uses $\EE[S_n]$ .

The exponential trick. For $t > 0$ , the function $x \mapsto e^{tx}$ is strictly increasing, so the event is unchanged by applying it to both sides. Apply Markov to the nonnegative random variable $e^{tS_n}$ :

\PP(S_n > 5n) = \PP\bigl(e^{tS_n} > e^{5nt}\bigr) \le \frac{\EE\bigl[e^{tS_n}\bigr]}{e^{5nt}}. \tag{8.16}

Why $e^{tx}$ ? I was confused why this particular function. We want to apply a function $T$ so that $\EE\bigl[T(\sum_i P_i)\bigr]$ splits into a product $\prod_i \EE[T(P_i)]$ by independence. That requires $T(x + y) = T(x)\,T(y)$ , which the exponential satisfies: $T(x + y) = e^{t(x+y)} = e^{tx}\, e^{ty} = T(x)\,T(y)$ . The parameter $t > 0$ is a scaling factor we are free to choose.

Since the $P_i$ are independent and identically distributed,

\EE\bigl[e^{tS_n}\bigr] = \EE\bigl[e^{tP_1 + tP_2 + \cdots + tP_n}\bigr] = \EE\bigl[e^{tP_1}\bigr]\,\EE\bigl[e^{tP_2}\bigr] \cdots \EE\bigl[e^{tP_n}\bigr] = \Bigl(\EE\bigl[e^{tP_1}\bigr]\Bigr)^n.

For $P \sim \Pois(\lambda)$ , using the Taylor series $e^{x} = \sum_{k \ge 0} \frac{x^k}{k!}$ ,

\EE\bigl[e^{tP}\bigr] = \sum_{k=0}^{\infty} e^{tk}\, \frac{e^{-\lambda}\lambda^k}{k!} = e^{-\lambda} \sum_{k=0}^{\infty} \frac{(e^{t}\lambda)^k}{k!} = e^{-\lambda}\, e^{e^t \lambda} = e^{\lambda(e^t - 1)}. \tag{8.18}

Substituting into (8.16),

\PP(S_n > 5n) \le \frac{\bigl(e^{\lambda(e^t - 1)}\bigr)^n}{e^{5nt}} = e^{\lambda n (e^t - 1) - 5nt} \overset{\lambda = 1}{=} e^{\,n\,(e^t - 1 - 5t)}.

Choosing $t$ . The bound holds for every $t > 0$ , so minimize the exponent $g(t) = e^t - 1 - 5t$ :

g'(t) = e^t - 5 = 0 \;\Longrightarrow\; t^\star = \ln 5, \qquad g''(t) = e^t > 0,

g(t^\star) = e^{\ln 5} - 1 - 5\ln 5 = 4 - 5\ln 5 \approx -4.05.

Therefore

\PP\Bigl(\sum_{i=1}^n P_i > 5n\Bigr) \le e^{-(5\ln 5 - 4)\,n} \approx e^{-4.05\,n}.

For $n = 10$ : Markov gives $0.2$ , while this bound gives $e^{-40.5} \approx 2.7 \times 10^{-18}$ .

Is the bound $\propto e^{-n}$ ? I was unsure about the rate. Any $t > 0$ with $g(t) < 0$ gives exponential decay $e^{n g(t)}$ ; the best rate is $5\ln 5 - 4 \approx 4.05$ , at $t^\star = \ln 5$ . This method (Markov applied to $e^{tX}$ , then optimizing over $t$ ) is called the Chernoff bound. The quantity $\EE[e^{tP}]$ is the MGF of $P$ , defined next.

Moment Generating Functions

Definition 8.2 (Moment generating function)

The moment generating function of a random variable $X$ is

M_X(s) \triangleq \EE\bigl[e^{sX}\bigr] = \begin{cases} \displaystyle \sum_x e^{sx}\, p_X(x) & \text{(discrete)}, \\[10pt] \displaystyle \int_{-\infty}^{\infty} e^{sx}\, f_X(x)\, dx & \text{(continuous)}, \end{cases}

for $s \in \RR$ (it may equal $\infty$ for some $s$ ).

MGFs are useful for:

tight tail bounds (Section 8.4);
computing all the moments $\EE[X^n]$ of a distribution;
sums of independent random variables: convolution becomes multiplication.

Theorem 8.3 (MGFs generate moments)

If $M_X(s) < \infty$ for all $s$ in an open interval around $0$ , then for every $n \ge 1$ ,

M_X^{(n)}(0) = \EE[X^n].

Proof.

Here $s$ is a constant inside the expectation and $X$ is the random part, so $\frac{d}{ds} e^{sX} = X e^{sX}$ . Differentiating $n$ times and moving the derivative inside the expectation (allowed because $M_X$ is finite near $0$ ),

\frac{d^n}{ds^n} M_X(s) = \frac{d^n}{ds^n} \EE\bigl[e^{sX}\bigr] = \EE\left[\frac{d^n}{ds^n} e^{sX}\right] = \EE\bigl[X^n e^{sX}\bigr].

At $s = 0$ , $e^{0 \cdot X} = e^0 = 1$ , so $M_X^{(n)}(0) = \EE\bigl[X^n e^{sX}\bigr]\big|_{s=0} = \EE[X^n]$ .

Example 8.4 (MGF of the exponential)

Let $X \sim \operatorname{Exponential}(\lambda)$ : $f_X(x) = \lambda e^{-\lambda x}$ for $x \in [0, \infty)$ . Then

M_X(s) = \int_0^{\infty} e^{sx}\, \lambda e^{-\lambda x}\, dx = \lambda \int_0^{\infty} e^{(s - \lambda)x}\, dx.

If $s < \lambda$ , the exponent $s - \lambda$ is negative and

M_X(s) = \lambda \left[\frac{1}{s - \lambda}\, e^{(s - \lambda)x}\right]_0^{\infty} = \lambda \left(0 - \frac{1}{s - \lambda}\right) = \frac{\lambda}{\lambda - s}.

If $s \ge \lambda$ , the integrand $e^{(s-\lambda)x} \ge 1$ for all $x \ge 0$ , so the integral diverges:

M_X(s) = \begin{cases} \dfrac{\lambda}{\lambda - s} & s < \lambda, \\[8pt] \infty & s \ge \lambda. \end{cases}

Moments. $M_X'(s) = \frac{\lambda}{(\lambda - s)^2}$ and $M_X''(s) = \frac{2\lambda}{(\lambda - s)^3}$ , so

\EE[X] = M_X'(0) = \frac{1}{\lambda}, \qquad \EE[X^2] = M_X''(0) = \frac{2}{\lambda^2}, \qquad \Var(X) = \frac{2}{\lambda^2} - \frac{1}{\lambda^2} = \frac{1}{\lambda^2}.

Theorem 8.4 (MGFs determine the distribution)

If $M_X(s) = M_Y(s) < \infty$ for all $s$ in some open interval $(-\varepsilon, \varepsilon)$ around $0$ , then $X$ and $Y$ have the same distribution.

So if we recognize an MGF, we know the distribution. (The proof is beyond the scope of this course.)

Not finished in lecture. We ran out of time after starting the list of MGF properties. The main properties are filled in below.

Theorem 8.5 (Properties of MGFs)

$M_X(0) = 1$ .
If $Y = aX + b$ , then $M_Y(s) = e^{sb}\, M_X(as)$ .
If $X$ and $Y$ are independent, then $M_{X+Y}(s) = M_X(s)\, M_Y(s)$ . More generally, for independent $X_1, \ldots, X_n$ , $M_{X_1 + \cdots + X_n}(s) = \prod_{i=1}^n M_{X_i}(s)$ , and $\bigl(M_{X_1}(s)\bigr)^n$ if they are also identically distributed.

Proof.

$M_X(0) = \EE\bigl[e^{0 \cdot X}\bigr] = \EE[1] = 1$ .
$M_Y(s) = \EE\bigl[e^{s(aX + b)}\bigr] = \EE\bigl[e^{sb}\, e^{(as)X}\bigr] = e^{sb}\, \EE\bigl[e^{(as)X}\bigr] = e^{sb}\, M_X(as)$ , since $e^{sb}$ is a constant.
$M_{X+Y}(s) = \EE\bigl[e^{s(X+Y)}\bigr] = \EE\bigl[e^{sX}\, e^{sY}\bigr] = \EE\bigl[e^{sX}\bigr]\,\EE\bigl[e^{sY}\bigr] = M_X(s)\, M_Y(s)$ , using independence of $e^{sX}$ and $e^{sY}$ . The $n$ -term version follows by induction.

Property 3 is the payoff: the density of $X + Y$ requires a convolution integral, but its MGF is just a product.

MGF Proofs for Sums of Independent Random Variables

Lemma 8.1 (MGF of the Gaussian)

If $X \sim \mathcal{N}(\mu, \sigma^2)$ , then for all $s \in \RR$ ,

M_X(s) = e^{\mu s + \sigma^2 s^2 / 2}.

Proof.

First let $Z \sim \mathcal{N}(0, 1)$ . Complete the square in the exponent, $z^2 - 2sz = (z - s)^2 - s^2$ :

\begin{aligned} M_Z(s) = \int_{-\infty}^{\infty} e^{sz}\, \frac{1}{\sqrt{2\pi}}\, e^{-z^2/2}\, dz &= \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, e^{-(z^2 - 2sz)/2}\, dz \\ &= \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, e^{-(z - s)^2/2 + s^2/2}\, dz \\ &= e^{s^2/2} \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, e^{-(z - s)^2/2}\, dz = e^{s^2/2} \cdot 1, \end{aligned}

since the last integrand is the $\mathcal{N}(s, 1)$ density. Now $X = \sigma Z + \mu$ (Lecture 7), so by Property 2 with $a = \sigma$ , $b = \mu$ ,

M_X(s) = e^{\mu s}\, M_Z(\sigma s) = e^{\mu s}\, e^{\sigma^2 s^2/2} = e^{\mu s + \sigma^2 s^2/2}.

Theorem 8.6 (Sum of independent Gaussians)

If $X \sim \mathcal{N}(\mu_X, \sigma_X^2)$ and $Y \sim \mathcal{N}(\mu_Y, \sigma_Y^2)$ are independent, then

X + Y \sim \mathcal{N}\bigl(\mu_X + \mu_Y,\; \sigma_X^2 + \sigma_Y^2\bigr).

Proof.

By Property 3 and the Gaussian MGF,

\begin{aligned} M_{X+Y}(s) = M_X(s)\, M_Y(s) &= e^{\mu_X s + \sigma_X^2 s^2/2} \cdot e^{\mu_Y s + \sigma_Y^2 s^2/2} \\ &= e^{(\mu_X + \mu_Y)s + (\sigma_X^2 + \sigma_Y^2)s^2/2}, \end{aligned}

which is the MGF of $\mathcal{N}(\mu_X + \mu_Y,\, \sigma_X^2 + \sigma_Y^2)$ , finite for all $s$ . Since MGFs determine the distribution, $X + Y \sim \mathcal{N}(\mu_X + \mu_Y,\, \sigma_X^2 + \sigma_Y^2)$ .

This completes the proof left as TBD in Lecture 7.

Corollary 8.1 (Sum of independent Poissons)

If $X \sim \Pois(\lambda)$ and $Y \sim \Pois(\mu)$ are independent, then $X + Y \sim \Pois(\lambda + \mu)$ .

Proof.

By (8.18), $M_X(s) = e^{\lambda(e^s - 1)}$ and $M_Y(s) = e^{\mu(e^s - 1)}$ , so

M_{X+Y}(s) = e^{\lambda(e^s - 1)}\, e^{\mu(e^s - 1)} = e^{(\lambda + \mu)(e^s - 1)},

the MGF of $\Pois(\lambda + \mu)$ . (Compare the convolution proof of Poisson merging in Lecture 4.)

Lecture 9: Concentration Inequalities

EECS 126 (UC Berkeley) · Spring 2026 · February 17, 2026

Reading: Bertsekas & Tsitsiklis, §5.1; supplementary notes on concentration inequalities.

These are my personal notes from Professor Kannan Ramchandran’s EECS 126 (Probability and Random Processes) course at UC Berkeley.

Topics: Markov’s inequality, Chebyshev’s inequality, the Chernoff bound, Jensen’s inequality, Hoeffding’s inequality, and a first look at the law of large numbers and the central limit theorem. Each bound trades more information about $X$ for a tighter estimate of a tail probability.

Markov’s Inequality

Theorem 9.1 (Markov’s inequality)

If $X \ge 0$ and $a > 0$ , then

\PP(X \ge a) \le \frac{\EE[X]}{a}. \tag{9.1}

Proof.

Compare $X$ with the random variable $a\,\mathbf{1}\{X \ge a\}$ pointwise:

if $X \ge a$ : $\;a\,\mathbf{1}\{X \ge a\} = a \le X$ ;
if $X < a$ : $\;a\,\mathbf{1}\{X \ge a\} = 0 \le X$ , since $X \ge 0$ .

So $X \ge a\,\mathbf{1}\{X \ge a\}$ always. Taking expectations, and using $\EE[\mathbf{1}_A] = \PP(A)$ ,

\EE[X] \ge a\,\EE\bigl[\mathbf{1}\{X \ge a\}\bigr] = a\,\PP(X \ge a).

Dividing by $a > 0$ gives (9.1).

The line x lies above the step function a·1{x ≥ a}, which jumps from 0 to a at x = a

For $x \ge 0$ , the line $x$ lies above the step $a\,\mathbf{1}\{x \ge a\}$ ; averaging both sides gives Markov.

Example 9.1 (Markov for an exponential)

Let $X \sim \operatorname{Exponential}(1)$ , so $\EE[X] = 1$ (Lecture 8) and $\PP(X \ge a) = \int_a^\infty e^{-x}\,dx = \Bigl[-e^{-x}\Bigr]_a^\infty = e^{-a}$ . Markov gives

\PP(X \ge a) \le \frac{1}{a}.

At $a = 5$ : Markov gives $0.2$ , while the exact value is $e^{-5} \approx 0.0067$ .

Markov cannot be improved using only the mean. Fix $m = \EE[X]$ and $a \ge m$ . The random variable

X = \begin{cases} a & \text{w.p. } \frac{m}{a}, \\[2pt] 0 & \text{w.p. } 1 - \frac{m}{a} \end{cases}

has $\EE[X] = a \cdot \frac{m}{a} = m$ and $\PP(X \ge a) = \frac{m}{a} = \frac{\EE[X]}{a}$ , so (9.1) holds with equality. To do better, we need more information about $X$ .

Chebyshev’s Inequality

Theorem 9.2 (Chebyshev’s inequality)

If $X$ has mean $\mu$ and variance $\sigma^2$ , then for every $c > 0$ ,

\PP\bigl(|X - \mu| \ge c\bigr) \le \frac{\sigma^2}{c^2}.

With $c = k\sigma$ :

\PP\bigl(|X - \mu| \ge k\sigma\bigr) \le \frac{1}{k^2}.

Proof.

Both sides of $|X - \mu| \ge c$ are nonnegative, so squaring does not change the event: $\{|X - \mu| \ge c\} = \{(X - \mu)^2 \ge c^2\}$ . Apply Markov to the nonnegative random variable $(X - \mu)^2$ with $a = c^2$ :

\PP\bigl(|X - \mu| \ge c\bigr) = \PP\bigl((X - \mu)^2 \ge c^2\bigr) \le \frac{\EE\bigl[(X - \mu)^2\bigr]}{c^2} = \frac{\sigma^2}{c^2}.

The parabola ((x − μ)/c)² lies above the indicator 1{|x − μ| ≥ c}, which is 1 outside μ ± c and 0 inside

The parabola $\bigl(\frac{x-\mu}{c}\bigr)^2$ lies above the indicator; averaging both sides gives Chebyshev. It uses the variance, not just the mean.

Example 9.2 (Chebyshev for a uniform)

Let $X \sim \operatorname{Uniform}[0,1]$ . Then $\mu = \frac12$ , $\EE[X^2] = \int_0^1 x^2\,dx = \frac13$ , and

\sigma^2 = \EE[X^2] - \mu^2 = \frac13 - \frac14 = \frac{1}{12}.

With $c = 0.4$ , the exact value is $\PP(|X - \tfrac12| \ge 0.4) = \PP(X \le 0.1) + \PP(X \ge 0.9) = 0.1 + 0.1 = 0.2$ , while

\PP\bigl(|X - \tfrac12| \ge 0.4\bigr) \le \frac{1/12}{(0.4)^2} = \frac{1}{12 \cdot 0.16} = \frac{1}{1.92} \approx 0.52.

Example 9.3 (Polling with Chebyshev)

To estimate the fraction $p$ of voters who support a candidate, poll $n$ people independently. Let $X_i \sim \Bern(p)$ indicate support and $M_n = \frac1n\sum_{i=1}^n X_i$ . How large must $n$ be so that $\PP(|M_n - p| \ge 0.01) \le 0.05$ ?

$\EE[M_n] = p$ and $\Var(M_n) = \frac{p(1-p)}{n}$ (Lecture 4 and Lecture 10). Since $p$ is unknown, bound the variance:

p(1 - p) = \frac14 - \Bigl(p - \frac12\Bigr)^2 \le \frac14.

By Chebyshev with $c = \varepsilon$ ,

\PP\bigl(|M_n - p| \ge \varepsilon\bigr) \le \frac{p(1-p)}{n\varepsilon^2} \le \frac{1}{4n\varepsilon^2}.

Setting $\frac{1}{4n\varepsilon^2} \le \delta$ with $\varepsilon = 0.01$ , $\delta = 0.05$ :

n \ge \frac{1}{4\varepsilon^2\delta} = \frac{1}{4\,(0.01)^2\,(0.05)} = \frac{1}{0.00002} = 50{,}000.

The Chernoff Bound

Theorem 9.3 (Chernoff bound)

For every $b$ and every $s > 0$ ,

\PP(X \ge b) \le e^{-sb}\, M_X(s), \qquad M_X(s) = \EE\bigl[e^{sX}\bigr],

and therefore

\PP(X \ge b) \le \min_{s > 0}\; e^{-sb}\, M_X(s).

For the lower tail, $\PP(X \le b) \le \min_{s < 0}\; e^{-sb}\, M_X(s)$ .

Proof.

For $s > 0$ , $x \mapsto e^{sx}$ is strictly increasing, so $\{X \ge b\} = \{e^{sX} \ge e^{sb}\}$ . Apply Markov to the nonnegative random variable $e^{sX}$ :

\PP(X \ge b) = \PP\bigl(e^{sX} \ge e^{sb}\bigr) \le \frac{\EE\bigl[e^{sX}\bigr]}{e^{sb}} = e^{-sb}\, M_X(s).

This holds for every $s > 0$ , so it holds for the minimizing $s$ . For $s < 0$ , $x \mapsto e^{sx}$ is strictly decreasing, so $\{X \le b\} = \{e^{sX} \ge e^{sb}\}$ , and the same argument gives the lower-tail bound.

The exponential e^{s(x − b)} lies above the indicator 1{x ≥ b}, which jumps from 0 to 1 at x = b

For any $s > 0$ , the exponential $e^{s(x-b)}$ lies above the indicator $\mathbf{1}\{x \ge b\}$ ; averaging gives $\PP(X \ge b) \le e^{-sb}M_X(s)$ , and we choose $s$ to make the curve hug the indicator as tightly as possible.

Example 9.4 (Gaussian tail)

Let $Z \sim \mathcal{N}(0,1)$ , so $M_Z(s) = e^{s^2/2}$ (Lecture 8). For $t > 0$ ,

\PP(Z \ge t) \le \min_{s > 0}\; e^{-st + s^2/2}.

The exponent $h(s) = \frac{s^2}{2} - st$ has $h'(s) = s - t = 0$ at $s^\star = t$ , and $h(t) = \frac{t^2}{2} - t^2 = -\frac{t^2}{2}$ . Therefore

\PP(Z \ge t) \le e^{-t^2/2}.

Compare with Chebyshev, which gives $\PP(Z \ge t) \le \PP(|Z| \ge t) \le \frac{1}{t^2}$ :

Log-scale plot for t from 1 to 4 of three curves: Chebyshev 1/t², Chernoff e^{−t²/2}, and the exact tail 1 − Φ(t)

Chebyshev decays polynomially; Chernoff decays like the true tail, up to a smaller factor.

Example 9.5 (Chernoff for coin flips)

Let $S_n \sim \Bin(n, p)$ , a sum of $n$ independent $\Bern(p)$ flips, so $M_{S_n}(s) = \bigl(1 - p + pe^s\bigr)^n$ : each flip has $\EE[e^{sX_i}] = (1-p)e^{0} + pe^{s}$ , and the MGF of an independent sum is the product (Lecture 8). For $p < q < 1$ ,

\PP(S_n \ge qn) \le \min_{s > 0}\; e^{-sqn}\bigl(1 - p + pe^s\bigr)^n = \min_{s > 0}\; \exp\Bigl(n\bigl[-sq + \ln(1 - p + pe^s)\bigr]\Bigr).

Set the derivative of the bracket to zero:

-q + \frac{pe^s}{1 - p + pe^s} = 0 \;\Longrightarrow\; pe^s = q(1 - p + pe^s) \;\Longrightarrow\; pe^s(1 - q) = q(1 - p) \;\Longrightarrow\; e^{s^\star} = \frac{q(1-p)}{p(1-q)},

and $s^\star > 0$ since $q > p$ . At $s^\star$ , $1 - p + pe^{s^\star} = 1 - p + \frac{q(1-p)}{1-q} = \frac{(1-p)(1 - q + q)}{1 - q} = \frac{1-p}{1-q}$ , so the bracket equals

\begin{aligned} -q\ln\frac{q(1-p)}{p(1-q)} + \ln\frac{1-p}{1-q} &= -q\ln\frac{q}{p} - q\ln\frac{1-p}{1-q} + \ln\frac{1-p}{1-q} \\ &= -q\ln\frac{q}{p} + (1 - q)\ln\frac{1-p}{1-q} \\ &= -\Bigl[q\ln\frac{q}{p} + (1 - q)\ln\frac{1-q}{1-p}\Bigr] \triangleq -D(q \,\|\, p). \end{aligned}

Therefore

\PP(S_n \ge qn) \le e^{-n\,D(q \| p)}.

Numbers. $n = 100$ fair flips ( $p = \frac12$ ), at least $75$ heads ( $q = \frac34$ ):

D\bigl(\tfrac34 \,\|\, \tfrac12\bigr) = \tfrac34\ln\tfrac{3/4}{1/2} + \tfrac14\ln\tfrac{1/4}{1/2} = \tfrac34\ln 1.5 + \tfrac14\ln 0.5 \approx 0.304 - 0.173 = 0.131.


Markov	$\frac{\EE[S_n]}{75} = \frac{50}{75}$	$\approx 0.67$
Chebyshev	$\PP(\lvert S_n - 50\rvert \ge 25) \le \frac{\Var(S_n)}{25^2} = \frac{25}{625}$	$= 0.04$
Chernoff	$e^{-100\,D(3/4 \Vert 1/2)} \approx e^{-13.1}$	$\approx 2.1 \times 10^{-6}$
Exact	$\sum_{k=75}^{100}\binom{100}{k}2^{-100}$	$\approx 2.8 \times 10^{-7}$

Jensen’s Inequality

Definition 9.1 (Convex function)

A function $g$ is convex if for all $x, y$ and $\lambda \in [0,1]$ ,

g\bigl(\lambda x + (1 - \lambda)y\bigr) \le \lambda g(x) + (1 - \lambda) g(y): \tag{9.23}

every chord lies above the graph. If $g$ is differentiable, convexity is equivalent to every tangent line lying below the graph:

g(x) \ge g(c) + g'(c)(x - c) \qquad \text{for all } x, c. \tag{9.24}

$g$ is concave if $-g$ is convex. Examples: $x^2$ , $e^{sx}$ , $|x|$ are convex; $\ln x$ and $\sqrt{x}$ are concave on $(0, \infty)$ .

Theorem 9.4 (Jensen’s inequality)

If $g$ is convex, then

g\bigl(\EE[X]\bigr) \le \EE\bigl[g(X)\bigr].

If $g$ is concave, the inequality reverses: $g\bigl(\EE[X]\bigr) \ge \EE\bigl[g(X)\bigr]$ .

Proof.

Take the tangent line (9.24) at $c = \EE[X]$ and substitute $x = X$ :

g(X) \ge g\bigl(\EE[X]\bigr) + g'\bigl(\EE[X]\bigr)\bigl(X - \EE[X]\bigr).

Take expectations. $g(\EE[X])$ and $g'(\EE[X])$ are constants, and $\EE\bigl[X - \EE[X]\bigr] = 0$ :

\EE\bigl[g(X)\bigr] \ge g\bigl(\EE[X]\bigr) + g'\bigl(\EE[X]\bigr) \cdot 0 = g\bigl(\EE[X]\bigr).

(For a convex $g$ that is not differentiable, the same argument works with any supporting line through $(\EE[X], g(\EE[X]))$ .) For concave $g$ , apply the result to $-g$ .

A convex curve g(x) with a chord between x_1 and x_2; at E[X] the chord's midpoint E[g(X)] lies above the curve's value g(E[X]), and the tangent at E[X] lies below the curve

For $X$ equal to $x_1$ or $x_2$ with probability $\frac12$ each, $\EE[g(X)]$ is the midpoint of the chord, which lies above $g(\EE[X])$ on the curve.

Example 9.6 (Consequences of Jensen)

$g(x) = x^2$ (convex): $\bigl(\EE[X]\bigr)^2 \le \EE[X^2]$ , i.e. $\Var(X) = \EE[X^2] - (\EE[X])^2 \ge 0$ .
$g(x) = e^{sx}$ (convex): $e^{s\EE[X]} \le M_X(s)$ , so an MGF is never smaller than $e^{s\EE[X]}$ .
$g(x) = \ln x$ (concave): $\EE[\ln X] \le \ln \EE[X]$ . For $X$ uniform on positive numbers $\{x_1, \ldots, x_n\}$ , $\frac1n \sum_{i=1}^n \ln x_i \le \ln\Bigl(\frac1n\sum_{i=1}^n x_i\Bigr) \;\Longrightarrow\; \Bigl(\prod_{i=1}^n x_i\Bigr)^{1/n} \le \frac1n \sum_{i=1}^n x_i,$ the arithmetic–geometric mean inequality.
In the St. Petersburg game (Lecture 5), $\EE[\log_2 W] = 2$ is finite even though $\EE[W] = \infty$ , consistent with $\EE[\log_2 W] \le \log_2 \EE[W]$ .

Hoeffding’s Inequality

Chernoff requires the MGF. For bounded random variables, we can bound the MGF without knowing the distribution.

Lemma 9.1 (Hoeffding’s lemma)

If $a \le X \le b$ and $\EE[X] = 0$ , then for every $s \in \RR$ ,

\EE\bigl[e^{sX}\bigr] \le e^{s^2 (b - a)^2 / 8}.

Proof.

Since $\EE[X] = 0$ , we have $a \le 0 \le b$ . Write $x \in [a, b]$ as the convex combination $x = \lambda a + (1 - \lambda) b$ with $\lambda = \frac{b - x}{b - a}$ . By convexity of $e^{sx}$ (9.23),

e^{sx} \le \frac{b - x}{b - a}\, e^{sa} + \frac{x - a}{b - a}\, e^{sb}.

Substitute $x = X$ and take expectations, using $\EE[X] = 0$ :

\EE\bigl[e^{sX}\bigr] \le \frac{b}{b - a}\, e^{sa} - \frac{a}{b - a}\, e^{sb}.

Let $p = \frac{-a}{b - a} \in [0, 1]$ and $h = s(b - a)$ , so $sa = -ph$ and $sb = h - ph$ . The right side becomes

(1 - p)\, e^{-ph} + p\, e^{h - ph} = e^{-ph}\bigl(1 - p + pe^h\bigr) = e^{L(h)}, \qquad L(h) = -ph + \ln\bigl(1 - p + pe^h\bigr).

Differentiate:

\begin{aligned} L'(h) &= -p + \frac{pe^h}{1 - p + pe^h}, \qquad L(0) = 0, \qquad L'(0) = -p + p = 0, \\ L''(h) &= \frac{pe^h(1 - p + pe^h) - pe^h \cdot pe^h}{(1 - p + pe^h)^2} = \frac{pe^h}{1 - p + pe^h}\cdot\frac{1 - p}{1 - p + pe^h} = u(1 - u) \le \frac14, \end{aligned}

where $u = \frac{pe^h}{1 - p + pe^h} \in [0,1]$ and $u(1-u) \le \frac14$ (as for $p(1-p)$ above). By Taylor’s theorem with remainder, for some $\xi$ between $0$ and $h$ ,

L(h) = L(0) + L'(0)\,h + \frac{L''(\xi)}{2}\, h^2 \le 0 + 0 + \frac{1}{8}\, h^2 = \frac{s^2(b - a)^2}{8}.

Theorem 9.5 (Hoeffding’s inequality)

Let $X_1, \ldots, X_n$ be independent with $a_i \le X_i \le b_i$ , and $S_n = \sum_{i=1}^n X_i$ . For every $t > 0$ ,

\PP\bigl(S_n - \EE[S_n] \ge t\bigr) \le \exp\Bigl(-\frac{2t^2}{\sum_{i=1}^n (b_i - a_i)^2}\Bigr).

In particular, if the $X_i$ are i.i.d. in $[a, b]$ with mean $\mu$ and $M_n = \frac{S_n}{n}$ ,

\PP\bigl(M_n - \mu \ge \varepsilon\bigr) \le e^{-2n\varepsilon^2/(b-a)^2}, \qquad \PP\bigl(|M_n - \mu| \ge \varepsilon\bigr) \le 2e^{-2n\varepsilon^2/(b-a)^2}. \tag{9.36}

Proof.

Let $C = \sum_i (b_i - a_i)^2$ . Each centered term $X_i - \EE[X_i]$ has mean $0$ and lies in an interval of length $b_i - a_i$ . By Chernoff, independence (Lecture 8), and Hoeffding’s lemma, for $s > 0$ ,

\begin{aligned} \PP\bigl(S_n - \EE[S_n] \ge t\bigr) &\le e^{-st}\, \EE\Bigl[e^{s\sum_i (X_i - \EE[X_i])}\Bigr] = e^{-st} \prod_{i=1}^n \EE\Bigl[e^{s(X_i - \EE[X_i])}\Bigr] \\ &\le e^{-st} \prod_{i=1}^n e^{s^2(b_i - a_i)^2/8} = \exp\Bigl(-st + \frac{s^2 C}{8}\Bigr). \end{aligned}

Minimize the exponent: $-t + \frac{sC}{4} = 0$ at $s^\star = \frac{4t}{C}$ , giving

-\frac{4t}{C}\, t + \frac{16t^2}{C^2}\cdot\frac{C}{8} = -\frac{4t^2}{C} + \frac{2t^2}{C} = -\frac{2t^2}{C}.

For the sample mean, $\{M_n - \mu \ge \varepsilon\} = \{S_n - n\mu \ge n\varepsilon\}$ , so take $t = n\varepsilon$ and $C = n(b-a)^2$ : $\frac{2t^2}{C} = \frac{2n^2\varepsilon^2}{n(b-a)^2} = \frac{2n\varepsilon^2}{(b-a)^2}$ . The lower tail is the same bound applied to $-X_i$ , and adding the two tails gives the factor $2$ .

Example 9.7 (Polling with Hoeffding)

Return to polling: $X_i \in [0, 1]$ , so $b - a = 1$ . By (9.36),

\PP\bigl(|M_n - p| \ge \varepsilon\bigr) \le 2e^{-2n\varepsilon^2} \le \delta \iff n \ge \frac{\ln(2/\delta)}{2\varepsilon^2}.

With $\varepsilon = 0.01$ , $\delta = 0.05$ :

n \ge \frac{\ln 40}{2\,(0.01)^2} = \frac{3.689}{0.0002} \approx 18{,}445,

compared with $50{,}000$ from Chebyshev. The two bounds as functions of $n$ (with $\varepsilon = 0.05$ ):

Log-scale plot for n from 100 to 2000: the Chebyshev bound 1/(4nε²) decays like 1/n while the Hoeffding bound 2e^{−2nε²} decays exponentially

Chebyshev decays like $\frac1n$ ; Hoeffding decays exponentially in $n$ .

A First Look at the LLN and the CLT

Law of large numbers. For i.i.d. $X_i$ with mean $\mu$ and variance $\sigma^2$ , Chebyshev gives

\PP\bigl(|M_n - \mu| \ge \varepsilon\bigr) \le \frac{\sigma^2}{n\varepsilon^2} \longrightarrow 0,

which is the weak law of large numbers (proved in full in Lecture 10).

Central limit theorem. $Z_n = \frac{\sqrt{n}\,(M_n - \mu)}{\sigma}$ is approximately $\mathcal{N}(0,1)$ for large $n$ (Lecture 10), so

\PP\bigl(|M_n - \mu| \ge \varepsilon\bigr) = \PP\Bigl(|Z_n| \ge \frac{\varepsilon\sqrt{n}}{\sigma}\Bigr) \approx 2\Bigl(1 - \Phi\Bigl(\frac{\varepsilon\sqrt{n}}{\sigma}\Bigr)\Bigr).

For polling, $\sigma^2 \le \frac14$ , and $2(1 - \Phi(1.96)) = 0.05$ , so we need $\frac{\varepsilon\sqrt{n}}{\sigma} \ge 1.96$ :

n \ge \frac{(1.96)^2\sigma^2}{\varepsilon^2}, \qquad\text{which holds for every } p \text{ once}\qquad n \ge \frac{(1.96)^2}{4\,(0.01)^2} = 9{,}604.

This is an approximation, not a guaranteed bound like Chebyshev or Hoeffding.

Method	Requires	Decay	Polling $n$
Markov	$X \ge 0$ , $\EE[X]$	$1/a$	—
Chebyshev	$\Var(X)$	$1/c^2$ , $1/n$	$50{,}000$
Chernoff	$M_X(s)$	exponential	—
Hoeffding	independent, bounded $X_i$	exponential in $n$	$18{,}445$
CLT	finite variance, large $n$	Gaussian (approximate)	$9{,}604$

Lecture 10: Laws of Large Numbers and the Central Limit Theorem

EECS 126 (UC Berkeley) · Spring 2026 · February 19, 2026

Reading: Bertsekas & Tsitsiklis, Ch. 5; supplementary notes on convergence.

These are my personal notes from Professor Kannan Ramchandran’s EECS 126 (Probability and Random Processes) course at UC Berkeley.

Topics: the weak law of large numbers (WLLN), convergence in probability, almost sure convergence and the strong law of large numbers (SLLN), and the central limit theorem (CLT).

Recap: Concentration Inequalities

The following bounds were derived in Lecture 9; we restate the ones used below.

Theorem 10.1 (Markov’s inequality)

If $X \ge 0$ and $a > 0$ , then

\PP(X \ge a) \le \frac{\EE[X]}{a}.

Theorem 10.2 (Chebyshev’s inequality)

If $X$ has mean $\mu$ and variance $\sigma^2$ , then for every $c > 0$ ,

\PP\bigl(|X - \mu| \ge c\bigr) \le \frac{\Var(X)}{c^2} = \frac{\sigma^2}{c^2}. \tag{10.2}

With $c = k\sigma$ :

\PP\bigl(|X - \mu| \ge k\sigma\bigr) \le \frac{1}{k^2}.

The left side is the probability that $X$ deviates from its mean by at least $c$ . For $k = 3$ , the two tails beyond $\mu \pm 3\sigma$ have total probability at most $\frac{1}{9}$ , for any distribution with finite variance.

A bell-shaped density with both tails beyond μ − 3σ and μ + 3σ shaded, labeled total tail area at most 1/9

Theorem 10.3 (Chernoff bound)

For every $b$ ,

\PP(X \ge b) \le \min_{s > 0}\; e^{-sb}\, M_X(s), \qquad M_X(s) = \EE\bigl[e^{sX}\bigr].

Example 10.1 (Chernoff bound for a Gaussian tail)

Let $X \sim \mathcal{N}(0, \sigma^2)$ , so $M_X(s) = e^{\sigma^2 s^2/2}$ (Lecture 8). With $b = k\sigma$ ,

\PP(X \ge k\sigma) \le \min_{s > 0}\; e^{-sk\sigma}\, e^{\sigma^2 s^2/2} = \min_{s > 0}\; e^{h(s)}, \qquad h(s) = \frac{\sigma^2 s^2}{2} - k\sigma s.

Minimize the exponent:

h'(s) = \sigma^2 s - k\sigma = 0 \;\Longrightarrow\; s^\star = \frac{k}{\sigma}, \qquad h(s^\star) = \frac{\sigma^2}{2}\cdot\frac{k^2}{\sigma^2} - k\sigma\cdot\frac{k}{\sigma} = \frac{k^2}{2} - k^2 = -\frac{k^2}{2}.

Therefore

\PP(X \ge k\sigma) \le e^{-k^2/2}.

Comparison at $k = 3$ (two-sided, $\PP(|X| \ge 3\sigma)$ ):


Chebyshev	$\frac{1}{k^2} = \frac{1}{9}$	$\approx 0.111$
Chernoff	$2e^{-k^2/2} = 2e^{-4.5}$	$\approx 0.0222$
Exact	$2\bigl(1 - \Phi(3)\bigr)$	$\approx 0.0027$

Chebyshev uses only the variance, so it holds for every distribution and is loose for the Gaussian. Chernoff uses the whole MGF and is much tighter.

The Weak Law of Large Numbers

Let $X_1, X_2, \ldots, X_n$ be i.i.d. with mean $\mu$ and variance $\sigma^2 < \infty$ . The sample mean (empirical mean estimate) is

M_n = \frac{1}{n} \sum_{i=1}^{n} X_i.

Theorem 10.4 (Weak law of large numbers)

For every $\varepsilon > 0$ ,

\PP\Bigl(\Bigl|\frac{1}{n}\sum_{i=1}^n X_i - \mu\Bigr| \ge \varepsilon\Bigr) \longrightarrow 0 \qquad \text{as } n \to \infty.

The probability that the sample mean deviates from the population mean by at least $\varepsilon$ vanishes as $n \to \infty$ .

Proof.

Compute the mean and variance of $M_n$ , then plug into Chebyshev. By linearity,

\EE[M_n] = \frac{1}{n} \sum_{i=1}^{n} \EE[X_i] = \frac{1}{n} \cdot n\mu = \mu.

By $\Var(aX) = a^2\Var(X)$ and independence (Lecture 4),

\Var(M_n) = \Var\Bigl(\frac{1}{n} \sum_{i=1}^{n} X_i\Bigr) = \Bigl(\frac{1}{n}\Bigr)^2 \Var\Bigl(\sum_{i=1}^{n} X_i\Bigr) = \Bigl(\frac{1}{n}\Bigr)^2 \cdot n\sigma^2 = \frac{\sigma^2}{n}.

By Chebyshev (10.2) with $X = M_n$ and $c = \varepsilon$ ,

\PP\bigl(|M_n - \mu| \ge \varepsilon\bigr) \le \frac{\Var(M_n)}{\varepsilon^2} = \frac{\sigma^2}{n\varepsilon^2} \longrightarrow 0 \qquad \text{as } n \to \infty. \tag{10.12}

Formally, for all $\varepsilon, \delta > 0$ there exists $n(\varepsilon, \delta)$ such that

\PP\bigl(|M_n - \mu| \ge \varepsilon\bigr) \le \delta \qquad \text{for all } n \ge n(\varepsilon, \delta).

From (10.12), $\frac{\sigma^2}{n\varepsilon^2} \le \delta \iff n \ge \frac{\sigma^2}{\varepsilon^2 \delta}$ , so $n(\varepsilon, \delta) = \frac{\sigma^2}{\varepsilon^2\delta}$ samples suffice. For example, with $\sigma^2 = 1$ , $\varepsilon = 0.1$ , $\delta = 0.05$ : $n \ge \frac{1}{(0.01)(0.05)} = 2000$ .

What are $\varepsilon$ and $\delta$ capturing? I had a question of what $\varepsilon$ captures in this statement.

$\varepsilon$ is the accuracy: how close $M_n$ must be to $\mu$ .
$\delta$ is the confidence: the probability of a deviation of at least $\varepsilon$ is at most $\delta$ . (It bounds a probability, not the deviation itself.)
$n(\varepsilon, \delta)$ is the number of samples needed for a given accuracy and confidence.

In words: for $n \ge n(\varepsilon, \delta)$ , $|M_n - \mu| < \varepsilon$ with probability at least $1 - \delta$ .

Convergence in Probability

Definition 10.1 (Convergence in probability)

A sequence $Y_1, Y_2, \ldots$ converges in probability to a constant $a$ , written $Y_n \xrightarrow{\;p\;} a$ , if for every $\varepsilon > 0$ ,

\lim_{n \to \infty} \PP\bigl(|Y_n - a| \ge \varepsilon\bigr) = 0.

The WLLN says exactly that $M_n \xrightarrow{\;p\;} \mu$ .

Example 10.2 (Minimum of uniforms)

Let $X_1, X_2, \ldots$ be i.i.d. $\operatorname{Uniform}[0,1]$ and $Y_n = \min(X_1, \ldots, X_n)$ . Does $Y_n$ converge in probability, and to what?

As we draw more samples, the smallest one should approach $0$ . Fix $0 < \varepsilon < 1$ . Since $Y_n \ge 0$ ,

\begin{aligned} \PP\bigl(|Y_n - 0| \ge \varepsilon\bigr) = \PP(Y_n \ge \varepsilon) &= \PP\bigl(\min(X_1, \ldots, X_n) \ge \varepsilon\bigr) \\ &= \PP(X_1 \ge \varepsilon,\, X_2 \ge \varepsilon,\, \ldots,\, X_n \ge \varepsilon) && \text{(all must be } \ge \varepsilon\text{)} \\ &= \prod_{i=1}^{n} \PP(X_i \ge \varepsilon) && \text{(independence)} \\ &= (1 - \varepsilon)^n \longrightarrow 0 \quad \text{as } n \to \infty, \end{aligned}

since $0 < 1 - \varepsilon < 1$ . (For $\varepsilon \ge 1$ the probability is $0$ .) Therefore $Y_n \xrightarrow{\;p\;} 0$ .

The interval [0, 1] with the part from ε to 1 shaded, labeled P(X_i ≥ ε) = 1 − ε

Example 10.3 (Discrete-time arrival process)

Time runs in slots $n = 1, 2, 3, \ldots$ , grouped into intervals

I_k = \{2^k,\, 2^k + 1,\, \ldots,\, 2^{k+1} - 1\}, \qquad k = 0, 1, 2, \ldots,

so $I_0 = \{1\}$ , $I_1 = \{2, 3\}$ , $I_2 = \{4, \ldots, 7\}$ , $I_3 = \{8, \ldots, 15\}$ , and $|I_k| = 2^k$ . There is exactly $1$ arrival in each interval $I_k$ , at a time chosen uniformly at random within $I_k$ . Let

Y_n = \begin{cases} 1 & \text{if there is an arrival at time } n, \\ 0 & \text{otherwise.} \end{cases}

Time slots 1 to 16 grouped into intervals I_0 = {1}, I_1 = {2, 3}, I_2 = {4, …, 7}, I_3 = {8, …, 15}, with one arrival marked in each interval

Each time in $I_k$ is equally likely to hold the arrival, so

\begin{aligned} \PP(Y_1 = 1) &= 1, \\ \PP(Y_2 = 1) &= \PP(Y_3 = 1) = \tfrac12, \\ \PP(Y_4 = 1) &= \cdots = \PP(Y_7 = 1) = \tfrac14, \end{aligned}

and in general

\PP(Y_n = 1) = \frac{1}{2^k} \qquad \text{if } n \in I_k.

As $n \to \infty$ , the interval index $k$ containing $n$ also goes to $\infty$ , so

\lim_{n \to \infty} \PP(Y_n = 1) = \lim_{k \to \infty} \frac{1}{2^k} = 0.

For $0 < \varepsilon \le 1$ , $\PP(|Y_n - 0| \ge \varepsilon) = \PP(Y_n = 1) \to 0$ (and it is $0$ for $\varepsilon > 1$ ), so $Y_n \xrightarrow{\;p\;} 0$ .

Converging, yet arrivals keep coming. Initially I was wondering how $Y_n$ can converge to $0$ when, no matter which $n$ I pick, infinitely many arrivals still come after it. This is why convergence in probability is a weak notion: it only says that at each fixed large $n$ , an arrival at that particular time is unlikely. It says nothing about a single realization of the whole sequence, which still has $Y_n = 1$ once in every interval. This weakness belongs to convergence in probability, not to the WLLN itself.

Almost Sure Convergence and the Strong Law

Each outcome $\omega$ of the experiment produces a whole realization: an ordinary sequence of numbers $Y_1(\omega), Y_2(\omega), \ldots$ .

Definition 10.2 (Almost sure convergence)

$Y_n$ converges almost surely (a.s.) to $Y$ , written $Y_n \xrightarrow{\;\text{a.s.}\;} Y$ , if

\PP\Bigl(\bigl\{\omega : \lim_{n \to \infty} Y_n(\omega) = Y(\omega)\bigr\}\Bigr) = 1.

That is, the realization converges as an ordinary sequence for every outcome outside an event of probability $0$ .

Theorem 10.5 (Strong law of large numbers)

Let $\{X_n\}_{n \ge 1}$ be i.i.d. with mean $\mu$ . Then

\PP\Bigl(\lim_{n \to \infty} \frac{1}{n} \sum_{i=1}^{n} X_i = \mu\Bigr) = 1, \qquad\text{i.e.}\qquad M_n \xrightarrow{\;\text{a.s.}\;} \mu.

(The proof is beyond the scope of this course.) For fair die rolls, $\mu = \frac{1 + 2 + \cdots + 6}{6} = \frac{21}{6} = 3.5$ , and every realization of $M_n$ (except on an event of probability $0$ ) converges to $3.5$ :

Three realizations of the running sample mean M_n of fair die rolls, each wandering early and then settling at 3.5

Excursions: SLLN versus WLLN. Initially I was wondering whether the SLLN allows any excursions at all outside a band $\mu \pm \varepsilon$ . It does: almost every realization leaves the band only finitely many times, and after some point stays inside forever. Convergence in probability (the WLLN) only requires that, at each fixed large $n$ , few realizations are outside the band; a single realization may still leave it infinitely often.

A realization that leaves the band μ ± ε a few times early on, then stays inside the band forever

Theorem 10.6 (Almost sure convergence implies convergence in probability)

If $Y_n \xrightarrow{\;\text{a.s.}\;} Y$ , then $Y_n \xrightarrow{\;p\;} Y$ . The converse is not necessarily true.

Proof.

Fix $\varepsilon > 0$ and let

A_n = \bigcup_{m \ge n} \bigl\{|Y_m - Y| \ge \varepsilon\bigr\},

the event that the realization leaves the $\varepsilon$ -band at some time $m \ge n$ . Then $A_1 \supseteq A_2 \supseteq \cdots$ . An outcome in every $A_n$ leaves the band infinitely often, so its realization does not converge to $Y$ ; hence $\PP\bigl(\bigcap_n A_n\bigr) \le \PP(Y_n \not\to Y) = 0$ . By continuity of probability for decreasing events (a consequence of countable additivity, Lecture 1), $\PP(A_n) \to \PP\bigl(\bigcap_n A_n\bigr) = 0$ . Since $\{|Y_n - Y| \ge \varepsilon\} \subseteq A_n$ ,

\PP\bigl(|Y_n - Y| \ge \varepsilon\bigr) \le \PP(A_n) \longrightarrow 0.

The converse fails: the arrival process. Every realization has exactly one arrival in each interval $I_k$ , so $Y_n(\omega) = 1$ for infinitely many $n$ and $Y_n(\omega) = 0$ for infinitely many $n$ . The sequence $Y_n(\omega)$ never converges, for every outcome $\omega$ , so $\PP(Y_n \to 0) = 0$ . Thus $Y_n \xrightarrow{\;p\;} 0$ but $Y_n$ does not converge almost surely.

The Central Limit Theorem

Let $X_1, X_2, \ldots$ be i.i.d. and $S_n = \sum_{i=1}^n X_i$ . What happens to the distribution of $S_n$ as $n$ grows?

Example 10.4 (Sums of uniforms)

$X_i \sim \operatorname{Uniform}[0,1]$ . $f_{S_1}$ is a box and $f_{S_2}$ is the triangle from Lecture 8. Convolving once more, $f_{S_3}(z) = \int_{z-1}^{z} f_{S_2}(x)\, dx$ :

\begin{aligned} 0 \le z \le 1: &\quad \int_0^z x\, dx = \frac{z^2}{2}, \\ 1 \le z \le 2: &\quad \int_{z-1}^{1} x\, dx + \int_1^z (2 - x)\, dx = \frac{1 - (z-1)^2}{2} + \Bigl(2z - \frac{z^2}{2} - \frac32\Bigr) = \frac{-2z^2 + 6z - 3}{2}, \\ 2 \le z \le 3: &\quad \int_{z-1}^{2} (2 - x)\, dx = \frac{(3 - z)^2}{2}. \end{aligned}

The densities of S_1 (a box on [0, 1]), S_2 (a triangle on [0, 2]) and S_3 (a smooth bell-like curve on [0, 3] centered at 1.5)

$f_{S_3}$ is symmetric about $1.5$ and already looks bell-shaped.

Example 10.5 (Sums of exponentials)

$X_i \sim \operatorname{Exponential}(\lambda)$ , so $f_{S_1}(x) = f_{X_1}(x) = \lambda e^{-\lambda x}$ for $x \ge 0$ . By convolution,

f_{S_2}(x) = \int_0^x \lambda e^{-\lambda u}\, \lambda e^{-\lambda (x - u)}\, du = \lambda^2 e^{-\lambda x} \int_0^x du = \lambda^2 x\, e^{-\lambda x}, \qquad x \ge 0.

By induction, $f_{S_n}(x) = \dfrac{\lambda^n x^{n-1} e^{-\lambda x}}{(n-1)!}$ :

f_{S_{n+1}}(x) = \int_0^x \frac{\lambda^n u^{n-1} e^{-\lambda u}}{(n-1)!}\, \lambda e^{-\lambda(x - u)}\, du = \frac{\lambda^{n+1} e^{-\lambda x}}{(n-1)!} \int_0^x u^{n-1}\, du = \frac{\lambda^{n+1} x^{n} e^{-\lambda x}}{n!}.

The densities of S_1 (a decaying exponential), S_2 and S_5, becoming more bell-shaped as n grows

Even starting from a very skewed density, $f_{S_n}$ becomes bell-shaped.

What happens to the mean and variance? By linearity and independence,

\EE[S_n] = n\mu, \qquad \Var(S_n) = n\sigma^2.

The variance always grows, and the mean grows too unless $\mu = 0$ , so $f_{S_n}$ drifts and spreads out. What about the sample mean $M_n = \frac{S_n}{n}$ ?

\EE[M_n] = \mu, \qquad \Var(M_n) = \frac{\sigma^2}{n} \longrightarrow 0,

so $f_{M_n}$ collapses to a spike at $\mu$ . Neither shows the shape of the limit:

A tall narrow spike f_{M_n} at μ next to a wide flat bell f_{S_n} centered at nμ with spread √n σ

So $M_n$ and $S_n$ are both the wrong objects. The right normalization subtracts the mean and divides by the standard deviation of $S_n$ :

Z_n = \frac{S_n - n\mu}{\sqrt{n}\,\sigma}.

Then

\EE[Z_n] = \frac{\EE[S_n] - n\mu}{\sqrt{n}\,\sigma} = \frac{n\mu - n\mu}{\sqrt{n}\,\sigma} = 0, \qquad \Var(Z_n) = \frac{\Var(S_n)}{\bigl(\sqrt{n}\,\sigma\bigr)^2} = \frac{n\sigma^2}{n\sigma^2} = 1.

Standardizing does not make $Z_n$ Gaussian. I had a question of whether the CLT just says that $Z_n$ becomes Gaussian once it is normalized. $\EE[Z_n] = 0$ and $\Var(Z_n) = 1$ for every $n$ , but for finite $n$ , $Z_n$ is generally not Gaussian (for uniform $X_i$ , $Z_n$ even has bounded support). The CLT is a statement about the limit: the CDF of $Z_n$ approaches $\Phi$ as $n \to \infty$ .

Definition 10.3 (Convergence in distribution)

$Z_n$ converges in distribution to $Z$ , written $Z_n \xrightarrow{\;d\;} Z$ , if

\lim_{n \to \infty} \PP(Z_n \le x) = \PP(Z \le x)

at every $x$ where the CDF of $Z$ is continuous.

Theorem 10.7 (Central limit theorem)

Let $X_1, X_2, \ldots$ be i.i.d. with mean $\mu$ and variance $0 < \sigma^2 < \infty$ , and $Z_n = \frac{S_n - n\mu}{\sqrt{n}\,\sigma}$ . Then for every $x \in \RR$ ,

\lim_{n \to \infty} \PP(Z_n \le x) = \Phi(x), \qquad \Phi(x) = \PP(Z \le x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-u^2/2}\, du,

where $Z \sim \mathcal{N}(0,1)$ and $u$ is a dummy variable. That is, $Z_n \xrightarrow{\;d\;} \mathcal{N}(0,1)$ .

Implications.

In the limit, the distribution of $Z_n$ “wipes out” all information about the distribution of the $X_i$ except the mean and the variance.
If there are a large number of independent small disturbances, their aggregate effect is approximately normally distributed. This is why the Gaussian is a good model for noise.

The finite-variance assumption is needed: without it, $Z_n$ need not approach a Gaussian.

Proof via MGFs.

Assume $M_{X_1}(s) < \infty$ for $s$ in an interval around $0$ . Standardize each term: $Y_i = \frac{X_i - \mu}{\sigma}$ , i.i.d. with $\EE[Y_i] = 0$ and $\EE[Y_i^2] = \Var(Y_i) = 1$ . Then

Z_n = \frac{S_n - n\mu}{\sqrt{n}\,\sigma} = \frac{\sum_{i=1}^n (X_i - \mu)}{\sqrt{n}\,\sigma} = \frac{1}{\sqrt{n}} \sum_{i=1}^{n} Y_i.

By the MGF properties of Lecture 8 (scaling, then independent sums),

M_{Z_n}(s) = \EE\Bigl[e^{\frac{s}{\sqrt{n}} \sum_{i} Y_i}\Bigr] = \prod_{i=1}^{n} \EE\Bigl[e^{\frac{s}{\sqrt{n}} Y_i}\Bigr] = \Bigl(M_Y\Bigl(\frac{s}{\sqrt{n}}\Bigr)\Bigr)^n.

Taylor-expand $M_Y$ around $0$ , using $M_Y^{(k)}(0) = \EE[Y^k]$ (Lecture 8):

M_Y(t) = M_Y(0) + M_Y'(0)\, t + \frac{M_Y''(0)}{2}\, t^2 + o(t^2) = 1 + \EE[Y]\, t + \frac{\EE[Y^2]}{2}\, t^2 + o(t^2) = 1 + \frac{t^2}{2} + o(t^2).

With $t = \frac{s}{\sqrt{n}}$ ,

M_Y\Bigl(\frac{s}{\sqrt{n}}\Bigr) = 1 + \frac{s^2}{2n} + o\Bigl(\frac{1}{n}\Bigr).

Take logarithms and use $\ln(1 + x) = x + O(x^2)$ as $x \to 0$ :

\ln M_{Z_n}(s) = n \ln\Bigl(1 + \frac{s^2}{2n} + o\Bigl(\frac{1}{n}\Bigr)\Bigr) = n \Bigl(\frac{s^2}{2n} + o\Bigl(\frac{1}{n}\Bigr)\Bigr) = \frac{s^2}{2} + o(1) \longrightarrow \frac{s^2}{2}.

So $M_{Z_n}(s) \to e^{s^2/2}$ , which is the MGF of $\mathcal{N}(0,1)$ (Lecture 8). Finally, we use without proof the continuity theorem: if $M_{Z_n}(s) \to M_Z(s)$ for all $s$ in an interval around $0$ , then $Z_n \xrightarrow{\;d\;} Z$ . Hence $Z_n \xrightarrow{\;d\;} \mathcal{N}(0,1)$ .

This proof assumes the MGF exists near $0$ . The CLT holds with only a finite variance; that more general proof uses characteristic functions and is beyond the scope of this course.