Lecture 10: Laws of Large Numbers and the Central Limit Theorem

EECS 126 (UC Berkeley) · Spring 2026 · February 19, 2026

Reading: Bertsekas & Tsitsiklis, Ch. 5; supplementary notes on convergence.

These are my personal notes from Professor Kannan Ramchandran’s EECS 126 (Probability and Random Processes) course at UC Berkeley.

Topics: the weak law of large numbers (WLLN), convergence in probability, almost sure convergence and the strong law of large numbers (SLLN), and the central limit theorem (CLT).

Recap: Concentration Inequalities

The following bounds were derived in Lecture 9; we restate the ones used below.

Theorem 10.1 (Markov’s inequality)

If $X \ge 0$ and $a > 0$ , then

\PP(X \ge a) \le \frac{\EE[X]}{a}.

Theorem 10.2 (Chebyshev’s inequality)

If $X$ has mean $\mu$ and variance $\sigma^2$ , then for every $c > 0$ ,

\PP\bigl(|X - \mu| \ge c\bigr) \le \frac{\Var(X)}{c^2} = \frac{\sigma^2}{c^2}. \tag{10.2}

With $c = k\sigma$ :

\PP\bigl(|X - \mu| \ge k\sigma\bigr) \le \frac{1}{k^2}.

The left side is the probability that $X$ deviates from its mean by at least $c$ . For $k = 3$ , the two tails beyond $\mu \pm 3\sigma$ have total probability at most $\frac{1}{9}$ , for any distribution with finite variance.

A bell-shaped density with both tails beyond μ − 3σ and μ + 3σ shaded, labeled total tail area at most 1/9

Theorem 10.3 (Chernoff bound)

For every $b$ ,

\PP(X \ge b) \le \min_{s > 0}\; e^{-sb}\, M_X(s), \qquad M_X(s) = \EE\bigl[e^{sX}\bigr].

Example 10.1 (Chernoff bound for a Gaussian tail)

Let $X \sim \mathcal{N}(0, \sigma^2)$ , so $M_X(s) = e^{\sigma^2 s^2/2}$ (Lecture 8). With $b = k\sigma$ ,

\PP(X \ge k\sigma) \le \min_{s > 0}\; e^{-sk\sigma}\, e^{\sigma^2 s^2/2} = \min_{s > 0}\; e^{h(s)}, \qquad h(s) = \frac{\sigma^2 s^2}{2} - k\sigma s.

Minimize the exponent:

h'(s) = \sigma^2 s - k\sigma = 0 \;\Longrightarrow\; s^\star = \frac{k}{\sigma}, \qquad h(s^\star) = \frac{\sigma^2}{2}\cdot\frac{k^2}{\sigma^2} - k\sigma\cdot\frac{k}{\sigma} = \frac{k^2}{2} - k^2 = -\frac{k^2}{2}.

Therefore

\PP(X \ge k\sigma) \le e^{-k^2/2}.

Comparison at $k = 3$ (two-sided, $\PP(|X| \ge 3\sigma)$ ):


Chebyshev	$\frac{1}{k^2} = \frac{1}{9}$	$\approx 0.111$
Chernoff	$2e^{-k^2/2} = 2e^{-4.5}$	$\approx 0.0222$
Exact	$2\bigl(1 - \Phi(3)\bigr)$	$\approx 0.0027$

Chebyshev uses only the variance, so it holds for every distribution and is loose for the Gaussian. Chernoff uses the whole MGF and is much tighter.

The Weak Law of Large Numbers

Let $X_1, X_2, \ldots, X_n$ be i.i.d. with mean $\mu$ and variance $\sigma^2 < \infty$ . The sample mean (empirical mean estimate) is

M_n = \frac{1}{n} \sum_{i=1}^{n} X_i.

Theorem 10.4 (Weak law of large numbers)

For every $\varepsilon > 0$ ,

\PP\Bigl(\Bigl|\frac{1}{n}\sum_{i=1}^n X_i - \mu\Bigr| \ge \varepsilon\Bigr) \longrightarrow 0 \qquad \text{as } n \to \infty.

The probability that the sample mean deviates from the population mean by at least $\varepsilon$ vanishes as $n \to \infty$ .

Proof.

Compute the mean and variance of $M_n$ , then plug into Chebyshev. By linearity,

\EE[M_n] = \frac{1}{n} \sum_{i=1}^{n} \EE[X_i] = \frac{1}{n} \cdot n\mu = \mu.

By $\Var(aX) = a^2\Var(X)$ and independence (Lecture 4),

\Var(M_n) = \Var\Bigl(\frac{1}{n} \sum_{i=1}^{n} X_i\Bigr) = \Bigl(\frac{1}{n}\Bigr)^2 \Var\Bigl(\sum_{i=1}^{n} X_i\Bigr) = \Bigl(\frac{1}{n}\Bigr)^2 \cdot n\sigma^2 = \frac{\sigma^2}{n}.

By Chebyshev (10.2) with $X = M_n$ and $c = \varepsilon$ ,

\PP\bigl(|M_n - \mu| \ge \varepsilon\bigr) \le \frac{\Var(M_n)}{\varepsilon^2} = \frac{\sigma^2}{n\varepsilon^2} \longrightarrow 0 \qquad \text{as } n \to \infty. \tag{10.12}

Formally, for all $\varepsilon, \delta > 0$ there exists $n(\varepsilon, \delta)$ such that

\PP\bigl(|M_n - \mu| \ge \varepsilon\bigr) \le \delta \qquad \text{for all } n \ge n(\varepsilon, \delta).

From (10.12), $\frac{\sigma^2}{n\varepsilon^2} \le \delta \iff n \ge \frac{\sigma^2}{\varepsilon^2 \delta}$ , so $n(\varepsilon, \delta) = \frac{\sigma^2}{\varepsilon^2\delta}$ samples suffice. For example, with $\sigma^2 = 1$ , $\varepsilon = 0.1$ , $\delta = 0.05$ : $n \ge \frac{1}{(0.01)(0.05)} = 2000$ .

What are $\varepsilon$ and $\delta$ capturing? I had a question of what $\varepsilon$ captures in this statement.

$\varepsilon$ is the accuracy: how close $M_n$ must be to $\mu$ .
$\delta$ is the confidence: the probability of a deviation of at least $\varepsilon$ is at most $\delta$ . (It bounds a probability, not the deviation itself.)
$n(\varepsilon, \delta)$ is the number of samples needed for a given accuracy and confidence.

In words: for $n \ge n(\varepsilon, \delta)$ , $|M_n - \mu| < \varepsilon$ with probability at least $1 - \delta$ .

Convergence in Probability

Definition 10.1 (Convergence in probability)

A sequence $Y_1, Y_2, \ldots$ converges in probability to a constant $a$ , written $Y_n \xrightarrow{\;p\;} a$ , if for every $\varepsilon > 0$ ,

\lim_{n \to \infty} \PP\bigl(|Y_n - a| \ge \varepsilon\bigr) = 0.

The WLLN says exactly that $M_n \xrightarrow{\;p\;} \mu$ .

Example 10.2 (Minimum of uniforms)

Let $X_1, X_2, \ldots$ be i.i.d. $\operatorname{Uniform}[0,1]$ and $Y_n = \min(X_1, \ldots, X_n)$ . Does $Y_n$ converge in probability, and to what?

As we draw more samples, the smallest one should approach $0$ . Fix $0 < \varepsilon < 1$ . Since $Y_n \ge 0$ ,

\begin{aligned} \PP\bigl(|Y_n - 0| \ge \varepsilon\bigr) = \PP(Y_n \ge \varepsilon) &= \PP\bigl(\min(X_1, \ldots, X_n) \ge \varepsilon\bigr) \\ &= \PP(X_1 \ge \varepsilon,\, X_2 \ge \varepsilon,\, \ldots,\, X_n \ge \varepsilon) && \text{(all must be } \ge \varepsilon\text{)} \\ &= \prod_{i=1}^{n} \PP(X_i \ge \varepsilon) && \text{(independence)} \\ &= (1 - \varepsilon)^n \longrightarrow 0 \quad \text{as } n \to \infty, \end{aligned}

since $0 < 1 - \varepsilon < 1$ . (For $\varepsilon \ge 1$ the probability is $0$ .) Therefore $Y_n \xrightarrow{\;p\;} 0$ .

The interval [0, 1] with the part from ε to 1 shaded, labeled P(X_i ≥ ε) = 1 − ε

Example 10.3 (Discrete-time arrival process)

Time runs in slots $n = 1, 2, 3, \ldots$ , grouped into intervals

I_k = \{2^k,\, 2^k + 1,\, \ldots,\, 2^{k+1} - 1\}, \qquad k = 0, 1, 2, \ldots,

so $I_0 = \{1\}$ , $I_1 = \{2, 3\}$ , $I_2 = \{4, \ldots, 7\}$ , $I_3 = \{8, \ldots, 15\}$ , and $|I_k| = 2^k$ . There is exactly $1$ arrival in each interval $I_k$ , at a time chosen uniformly at random within $I_k$ . Let

Y_n = \begin{cases} 1 & \text{if there is an arrival at time } n, \\ 0 & \text{otherwise.} \end{cases}

Time slots 1 to 16 grouped into intervals I_0 = {1}, I_1 = {2, 3}, I_2 = {4, …, 7}, I_3 = {8, …, 15}, with one arrival marked in each interval

Each time in $I_k$ is equally likely to hold the arrival, so

\begin{aligned} \PP(Y_1 = 1) &= 1, \\ \PP(Y_2 = 1) &= \PP(Y_3 = 1) = \tfrac12, \\ \PP(Y_4 = 1) &= \cdots = \PP(Y_7 = 1) = \tfrac14, \end{aligned}

and in general

\PP(Y_n = 1) = \frac{1}{2^k} \qquad \text{if } n \in I_k.

As $n \to \infty$ , the interval index $k$ containing $n$ also goes to $\infty$ , so

\lim_{n \to \infty} \PP(Y_n = 1) = \lim_{k \to \infty} \frac{1}{2^k} = 0.

For $0 < \varepsilon \le 1$ , $\PP(|Y_n - 0| \ge \varepsilon) = \PP(Y_n = 1) \to 0$ (and it is $0$ for $\varepsilon > 1$ ), so $Y_n \xrightarrow{\;p\;} 0$ .

Converging, yet arrivals keep coming. Initially I was wondering how $Y_n$ can converge to $0$ when, no matter which $n$ I pick, infinitely many arrivals still come after it. This is why convergence in probability is a weak notion: it only says that at each fixed large $n$ , an arrival at that particular time is unlikely. It says nothing about a single realization of the whole sequence, which still has $Y_n = 1$ once in every interval. This weakness belongs to convergence in probability, not to the WLLN itself.

Almost Sure Convergence and the Strong Law

Each outcome $\omega$ of the experiment produces a whole realization: an ordinary sequence of numbers $Y_1(\omega), Y_2(\omega), \ldots$ .

Definition 10.2 (Almost sure convergence)

$Y_n$ converges almost surely (a.s.) to $Y$ , written $Y_n \xrightarrow{\;\text{a.s.}\;} Y$ , if

\PP\Bigl(\bigl\{\omega : \lim_{n \to \infty} Y_n(\omega) = Y(\omega)\bigr\}\Bigr) = 1.

That is, the realization converges as an ordinary sequence for every outcome outside an event of probability $0$ .

Theorem 10.5 (Strong law of large numbers)

Let $\{X_n\}_{n \ge 1}$ be i.i.d. with mean $\mu$ . Then

\PP\Bigl(\lim_{n \to \infty} \frac{1}{n} \sum_{i=1}^{n} X_i = \mu\Bigr) = 1, \qquad\text{i.e.}\qquad M_n \xrightarrow{\;\text{a.s.}\;} \mu.

(The proof is beyond the scope of this course.) For fair die rolls, $\mu = \frac{1 + 2 + \cdots + 6}{6} = \frac{21}{6} = 3.5$ , and every realization of $M_n$ (except on an event of probability $0$ ) converges to $3.5$ :

Three realizations of the running sample mean M_n of fair die rolls, each wandering early and then settling at 3.5

Excursions: SLLN versus WLLN. Initially I was wondering whether the SLLN allows any excursions at all outside a band $\mu \pm \varepsilon$ . It does: almost every realization leaves the band only finitely many times, and after some point stays inside forever. Convergence in probability (the WLLN) only requires that, at each fixed large $n$ , few realizations are outside the band; a single realization may still leave it infinitely often.

A realization that leaves the band μ ± ε a few times early on, then stays inside the band forever

Theorem 10.6 (Almost sure convergence implies convergence in probability)

If $Y_n \xrightarrow{\;\text{a.s.}\;} Y$ , then $Y_n \xrightarrow{\;p\;} Y$ . The converse is not necessarily true.

Proof.

Fix $\varepsilon > 0$ and let

A_n = \bigcup_{m \ge n} \bigl\{|Y_m - Y| \ge \varepsilon\bigr\},

the event that the realization leaves the $\varepsilon$ -band at some time $m \ge n$ . Then $A_1 \supseteq A_2 \supseteq \cdots$ . An outcome in every $A_n$ leaves the band infinitely often, so its realization does not converge to $Y$ ; hence $\PP\bigl(\bigcap_n A_n\bigr) \le \PP(Y_n \not\to Y) = 0$ . By continuity of probability for decreasing events (a consequence of countable additivity, Lecture 1), $\PP(A_n) \to \PP\bigl(\bigcap_n A_n\bigr) = 0$ . Since $\{|Y_n - Y| \ge \varepsilon\} \subseteq A_n$ ,

\PP\bigl(|Y_n - Y| \ge \varepsilon\bigr) \le \PP(A_n) \longrightarrow 0.

The converse fails: the arrival process. Every realization has exactly one arrival in each interval $I_k$ , so $Y_n(\omega) = 1$ for infinitely many $n$ and $Y_n(\omega) = 0$ for infinitely many $n$ . The sequence $Y_n(\omega)$ never converges, for every outcome $\omega$ , so $\PP(Y_n \to 0) = 0$ . Thus $Y_n \xrightarrow{\;p\;} 0$ but $Y_n$ does not converge almost surely.

The Central Limit Theorem

Let $X_1, X_2, \ldots$ be i.i.d. and $S_n = \sum_{i=1}^n X_i$ . What happens to the distribution of $S_n$ as $n$ grows?

Example 10.4 (Sums of uniforms)

$X_i \sim \operatorname{Uniform}[0,1]$ . $f_{S_1}$ is a box and $f_{S_2}$ is the triangle from Lecture 8. Convolving once more, $f_{S_3}(z) = \int_{z-1}^{z} f_{S_2}(x)\, dx$ :

\begin{aligned} 0 \le z \le 1: &\quad \int_0^z x\, dx = \frac{z^2}{2}, \\ 1 \le z \le 2: &\quad \int_{z-1}^{1} x\, dx + \int_1^z (2 - x)\, dx = \frac{1 - (z-1)^2}{2} + \Bigl(2z - \frac{z^2}{2} - \frac32\Bigr) = \frac{-2z^2 + 6z - 3}{2}, \\ 2 \le z \le 3: &\quad \int_{z-1}^{2} (2 - x)\, dx = \frac{(3 - z)^2}{2}. \end{aligned}

The densities of S_1 (a box on [0, 1]), S_2 (a triangle on [0, 2]) and S_3 (a smooth bell-like curve on [0, 3] centered at 1.5)

$f_{S_3}$ is symmetric about $1.5$ and already looks bell-shaped.

Example 10.5 (Sums of exponentials)

$X_i \sim \operatorname{Exponential}(\lambda)$ , so $f_{S_1}(x) = f_{X_1}(x) = \lambda e^{-\lambda x}$ for $x \ge 0$ . By convolution,

f_{S_2}(x) = \int_0^x \lambda e^{-\lambda u}\, \lambda e^{-\lambda (x - u)}\, du = \lambda^2 e^{-\lambda x} \int_0^x du = \lambda^2 x\, e^{-\lambda x}, \qquad x \ge 0.

By induction, $f_{S_n}(x) = \dfrac{\lambda^n x^{n-1} e^{-\lambda x}}{(n-1)!}$ :

f_{S_{n+1}}(x) = \int_0^x \frac{\lambda^n u^{n-1} e^{-\lambda u}}{(n-1)!}\, \lambda e^{-\lambda(x - u)}\, du = \frac{\lambda^{n+1} e^{-\lambda x}}{(n-1)!} \int_0^x u^{n-1}\, du = \frac{\lambda^{n+1} x^{n} e^{-\lambda x}}{n!}.

The densities of S_1 (a decaying exponential), S_2 and S_5, becoming more bell-shaped as n grows

Even starting from a very skewed density, $f_{S_n}$ becomes bell-shaped.

What happens to the mean and variance? By linearity and independence,

\EE[S_n] = n\mu, \qquad \Var(S_n) = n\sigma^2.

The variance always grows, and the mean grows too unless $\mu = 0$ , so $f_{S_n}$ drifts and spreads out. What about the sample mean $M_n = \frac{S_n}{n}$ ?

\EE[M_n] = \mu, \qquad \Var(M_n) = \frac{\sigma^2}{n} \longrightarrow 0,

so $f_{M_n}$ collapses to a spike at $\mu$ . Neither shows the shape of the limit:

A tall narrow spike f_{M_n} at μ next to a wide flat bell f_{S_n} centered at nμ with spread √n σ

So $M_n$ and $S_n$ are both the wrong objects. The right normalization subtracts the mean and divides by the standard deviation of $S_n$ :

Z_n = \frac{S_n - n\mu}{\sqrt{n}\,\sigma}.

Then

\EE[Z_n] = \frac{\EE[S_n] - n\mu}{\sqrt{n}\,\sigma} = \frac{n\mu - n\mu}{\sqrt{n}\,\sigma} = 0, \qquad \Var(Z_n) = \frac{\Var(S_n)}{\bigl(\sqrt{n}\,\sigma\bigr)^2} = \frac{n\sigma^2}{n\sigma^2} = 1.

Standardizing does not make $Z_n$ Gaussian. I had a question of whether the CLT just says that $Z_n$ becomes Gaussian once it is normalized. $\EE[Z_n] = 0$ and $\Var(Z_n) = 1$ for every $n$ , but for finite $n$ , $Z_n$ is generally not Gaussian (for uniform $X_i$ , $Z_n$ even has bounded support). The CLT is a statement about the limit: the CDF of $Z_n$ approaches $\Phi$ as $n \to \infty$ .

Definition 10.3 (Convergence in distribution)

$Z_n$ converges in distribution to $Z$ , written $Z_n \xrightarrow{\;d\;} Z$ , if

\lim_{n \to \infty} \PP(Z_n \le x) = \PP(Z \le x)

at every $x$ where the CDF of $Z$ is continuous.

Theorem 10.7 (Central limit theorem)

Let $X_1, X_2, \ldots$ be i.i.d. with mean $\mu$ and variance $0 < \sigma^2 < \infty$ , and $Z_n = \frac{S_n - n\mu}{\sqrt{n}\,\sigma}$ . Then for every $x \in \RR$ ,

\lim_{n \to \infty} \PP(Z_n \le x) = \Phi(x), \qquad \Phi(x) = \PP(Z \le x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-u^2/2}\, du,

where $Z \sim \mathcal{N}(0,1)$ and $u$ is a dummy variable. That is, $Z_n \xrightarrow{\;d\;} \mathcal{N}(0,1)$ .

Implications.

In the limit, the distribution of $Z_n$ “wipes out” all information about the distribution of the $X_i$ except the mean and the variance.
If there are a large number of independent small disturbances, their aggregate effect is approximately normally distributed. This is why the Gaussian is a good model for noise.

The finite-variance assumption is needed: without it, $Z_n$ need not approach a Gaussian.

Proof via MGFs.

Assume $M_{X_1}(s) < \infty$ for $s$ in an interval around $0$ . Standardize each term: $Y_i = \frac{X_i - \mu}{\sigma}$ , i.i.d. with $\EE[Y_i] = 0$ and $\EE[Y_i^2] = \Var(Y_i) = 1$ . Then

Z_n = \frac{S_n - n\mu}{\sqrt{n}\,\sigma} = \frac{\sum_{i=1}^n (X_i - \mu)}{\sqrt{n}\,\sigma} = \frac{1}{\sqrt{n}} \sum_{i=1}^{n} Y_i.

By the MGF properties of Lecture 8 (scaling, then independent sums),

M_{Z_n}(s) = \EE\Bigl[e^{\frac{s}{\sqrt{n}} \sum_{i} Y_i}\Bigr] = \prod_{i=1}^{n} \EE\Bigl[e^{\frac{s}{\sqrt{n}} Y_i}\Bigr] = \Bigl(M_Y\Bigl(\frac{s}{\sqrt{n}}\Bigr)\Bigr)^n.

Taylor-expand $M_Y$ around $0$ , using $M_Y^{(k)}(0) = \EE[Y^k]$ (Lecture 8):

M_Y(t) = M_Y(0) + M_Y'(0)\, t + \frac{M_Y''(0)}{2}\, t^2 + o(t^2) = 1 + \EE[Y]\, t + \frac{\EE[Y^2]}{2}\, t^2 + o(t^2) = 1 + \frac{t^2}{2} + o(t^2).

With $t = \frac{s}{\sqrt{n}}$ ,

M_Y\Bigl(\frac{s}{\sqrt{n}}\Bigr) = 1 + \frac{s^2}{2n} + o\Bigl(\frac{1}{n}\Bigr).

Take logarithms and use $\ln(1 + x) = x + O(x^2)$ as $x \to 0$ :

\ln M_{Z_n}(s) = n \ln\Bigl(1 + \frac{s^2}{2n} + o\Bigl(\frac{1}{n}\Bigr)\Bigr) = n \Bigl(\frac{s^2}{2n} + o\Bigl(\frac{1}{n}\Bigr)\Bigr) = \frac{s^2}{2} + o(1) \longrightarrow \frac{s^2}{2}.

So $M_{Z_n}(s) \to e^{s^2/2}$ , which is the MGF of $\mathcal{N}(0,1)$ (Lecture 8). Finally, we use without proof the continuity theorem: if $M_{Z_n}(s) \to M_Z(s)$ for all $s$ in an interval around $0$ , then $Z_n \xrightarrow{\;d\;} Z$ . Hence $Z_n \xrightarrow{\;d\;} \mathcal{N}(0,1)$ .

This proof assumes the MGF exists near $0$ . The CLT holds with only a finite variance; that more general proof uses characteristic functions and is beyond the scope of this course.

laws-of-large-numbers-and-the-central-limit-theorem.html