Lecture 8: Sums of Independent Random Variables and Moment Generating Functions

EECS 126 (UC Berkeley) · Spring 2026 · February 12, 2026

Reading: Bertsekas & Tsitsiklis, §4.4–4.6.

These are my personal notes from Professor Kannan Ramchandran’s EECS 126 (Probability and Random Processes) course at UC Berkeley.

Topics: sums of independent random variables (convolution), tail bounds, and moment generating functions (MGFs).

Discrete Convolution

Example 8.1 (Sum of two independent discrete random variables)

Let $X$ and $Y$ be independent, each taking values in $\{0, 1, 2, 3\}$ , with

p_X = (0.1,\ 0.2,\ 0.3,\ 0.4), \qquad p_Y = (0.4,\ 0.3,\ 0.2,\ 0.1)

at the values $0, 1, 2, 3$ respectively. Let $Z = X + Y$ . Find $\PP(Z = 3)$ .

Bar charts of the PMFs: p_X = 0.1, 0.2, 0.3, 0.4 and p_Y = 0.4, 0.3, 0.2, 0.1 at the values 0, 1, 2, 3

$Z = 3$ occurs exactly for the pairs $(x, y) \in \{(0,3), (1,2), (2,1), (3,0)\}$ , i.e. $y = 3 - x$ :

\begin{aligned} \PP(Z = 3) = \PP(X + Y = 3) &= \sum_{(x,y)\,:\, x + y = 3} \PP(X = x,\, Y = y) \\ &= \sum_{(x,y)\,:\, x + y = 3} \PP(X = x)\, \PP(Y = y) \qquad \text{(independence)} \\ &= \sum_{x} \PP(X = x)\, \PP(Y = 3 - x) \\ &= p_X(0)\,p_Y(3) + p_X(1)\,p_Y(2) + p_X(2)\,p_Y(1) + p_X(3)\,p_Y(0) \\ &= (0.1)(0.1) + (0.2)(0.2) + (0.3)(0.3) + (0.4)(0.4) \\ &= 0.01 + 0.04 + 0.09 + 0.16 = 0.30. \end{aligned}

To see the products line up, flip $p_Y$ and shift it by $3$ : plot $p_Y(3 - x)$ against $x$ , directly under $p_X(x)$ , then multiply bar by bar and add.

p_X(x) above the flipped and shifted p_Y(3 − x); multiplying bar by bar gives 0.01 + 0.04 + 0.09 + 0.16 = 0.30 = P(Z = 3)

Definition 8.1 (Discrete convolution)

If $X$ and $Y$ are independent and integer-valued, then $Z = X + Y$ has PMF

p_Z(z) = (p_X * p_Y)(z) \triangleq \sum_{x} p_X(x)\, p_Y(z - x). \tag{8.3}

Here $x$ is a dummy summation variable.

Applying (8.3) at every $z$ in the example gives the full PMF of $Z$ :

Bar chart of the PMF of Z = X + Y: 0.04, 0.11, 0.20, 0.30, 0.20, 0.11, 0.04 at z = 0 through 6, with z = 3 highlighted

Check: $0.04 + 0.11 + 0.20 + 0.30 + 0.20 + 0.11 + 0.04 = 1$ .

Continuous Convolution

Theorem 8.1 (Density of a sum of independent random variables)

If $X$ and $Y$ are independent continuous random variables and $Z = X + Y$ , then

f_Z(z) = (f_X * f_Y)(z) \triangleq \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, dx. \tag{8.4}

Here $x$ is a dummy variable of integration.

Proof by conditioning.

Find the conditional CDF of $Z$ given $X = x$ , then differentiate:

\begin{aligned} F_{Z \mid X}(z \mid x) = \PP(Z \le z \mid X = x) &= \PP(X + Y \le z \mid X = x) \\ &= \PP(x + Y \le z \mid X = x) \\ &= \PP(Y \le z - x \mid X = x) \\ &= \PP(Y \le z - x) && \text{(independence)} \\ &= F_Y(z - x). \end{aligned}

Differentiating in $z$ ,

f_{Z \mid X}(z \mid x) = \frac{d}{dz} F_Y(z - x) = f_Y(z - x).

Multiply both sides by the marginal $f_X(x)$ ; the left side becomes the joint PDF of $X$ and $Z$ :

f_{X,Z}(x, z) = f_{Z \mid X}(z \mid x)\, f_X(x) = f_Y(z - x)\, f_X(x).

To get the marginal $f_Z$ , integrate over all possible $x$ :

f_Z(z) = \int_{-\infty}^{\infty} f_{X,Z}(x, z)\, dx = \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, dx.

Why does the conditioning drop? I was unsure why we can remove “ $\mid X = x$ ”. Once $X = x$ is fixed, the only random variable left in $\{Y \le z - x\}$ is $Y$ , and $Y$ is independent of $X$ , so conditioning on $X = x$ does not change its distribution.

Proof by a thin strip.

$\PP(z \le Z \le z + \delta)$ is the probability that $(X, Y)$ lands in the strip between the lines $y = z - x$ and $y = z + \delta - x$ :

\begin{aligned} \PP(z \le Z \le z + \delta) &= \int_{-\infty}^{\infty} \int_{z - x}^{z - x + \delta} f_{X,Y}(x, y)\, dy\, dx \\ &= \int_{-\infty}^{\infty} \int_{z - x}^{z - x + \delta} f_X(x)\, f_Y(y)\, dy\, dx && \text{(independence)} \\ &= \int_{-\infty}^{\infty} f_X(x) \left[\int_{z - x}^{z - x + \delta} f_Y(y)\, dy\right] dx \\ &\approx \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, \delta\, dx, \end{aligned}

where the inner integral over an interval of width $\delta$ is approximated by a rectangle of height $f_Y(z - x)$ . Since also $\PP(z \le Z \le z + \delta) \approx f_Z(z)\,\delta$ ,

f_Z(z)\, \delta \approx \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, \delta\, dx,

and dividing by $\delta$ and letting $\delta \to 0$ gives (8.4).

The thin strip between the lines x + y = z and x + y = z + δ in the (x, y) plane

The strip has negative slope because $Z = X + Y$ means $Y = Z - X$ . What we integrate over the strip is the joint density, so $\PP(z \le Z \le z+\delta)$ is the probability mass of the strip, not its area.

Example: Sum of Two Uniforms

Example 8.2 (Sum of two independent uniforms)

Let $X, Y \sim \operatorname{Uniform}[0,1]$ be independent, so $f_X(x) = \frac{1}{b - a} = \frac{1}{1 - 0} = 1$ on $[0,1]$ , and the same for $f_Y$ . Find $f_Z$ for $Z = X + Y$ .

By (8.4), writing each density as an indicator,

f_Z(z) = \int_{-\infty}^{\infty} f_X(x)\, f_Y(z - x)\, dx = \int_{-\infty}^{\infty} \mathbf{1}\{0 \le x \le 1\} \cdot \mathbf{1}\{0 \le z - x \le 1\}\, dx.

The second indicator is $1$ when $z - 1 \le x \le z$ . So the integrand is $1$ exactly on $\bigl[\max(0, z - 1),\, \min(1, z)\bigr]$ , and $f_Z(z)$ is the length of that interval:

\begin{aligned} 0 \le z \le 1: &\quad f_Z(z) = \int_0^{z} 1\, dx = z, \\ 1 \le z \le 2: &\quad f_Z(z) = \int_{z-1}^{1} 1\, dx = 1 - (z - 1) = 2 - z, \end{aligned}

and $f_Z(z) = 0$ otherwise. Check: the triangle has area $\tfrac12 \cdot 2 \cdot 1 = 1$ .

Left: the box f_X(x) on [0, 1] overlapping the sliding box f_Y(z − x) on [z − 1, z]. Right: the triangular density f_Z(z) on [0, 2] with peak 1 at z = 1

Sliding overlap. As a function of $x$ , $f_Y(z - x)$ is a box on $[z - 1, z]$ that slides to the right as $z$ increases. $f_Z(z)$ is the length of its overlap with the box $[0,1]$ : it grows from $0$ at $z = 0$ , is largest at $z = 1$ (exact overlap), and shrinks back to $0$ at $z = 2$ . The same flip-and-slide picture is how convolution is used in signal processing.

Bounding Tail Probabilities

Example 8.3 (How big must a buffer be?)

Over $n$ time slots, $P_i$ packets arrive in slot $i$ , where $P_1, \ldots, P_n \sim \Pois(1)$ are i.i.d. A buffer holds $5n$ packets. How likely is it to overflow? We want to bound

\PP\Bigl(\sum_{i=1}^{n} P_i > 5n\Bigr).

Theorem 8.2 (Markov’s inequality)

If $X \ge 0$ and $a > 0$ , then

\PP(X \ge a) \le \frac{\EE[X]}{a}.

Proof.

Pointwise, $X \ge a\,\mathbf{1}\{X \ge a\}$ : if $X \ge a$ the right side is $a \le X$ , and otherwise it is $0 \le X$ . Taking expectations, $\EE[X] \ge a\,\EE\bigl[\mathbf{1}\{X \ge a\}\bigr] = a\,\PP(X \ge a)$ .

Let $S_n = \sum_{i=1}^n P_i \ge 0$ . By linearity of expectation and $\EE[P_i] = 1$ for $\Pois(1)$ ,

\PP(S_n > 5n) \le \PP(S_n \ge 5n) \le \frac{\EE[S_n]}{5n} = \frac{\sum_{i=1}^n \EE[P_i]}{5n} = \frac{\sum_{i=1}^n 1}{5n} = \frac{n}{5n} = \frac15.

Too loose. The bound $\frac15$ does not shrink as $n$ grows, even though $S_n/n$ stays close to its mean $1$ , far below $5$ , for large $n$ . Markov only uses $\EE[S_n]$ .

The exponential trick. For $t > 0$ , the function $x \mapsto e^{tx}$ is strictly increasing, so the event is unchanged by applying it to both sides. Apply Markov to the nonnegative random variable $e^{tS_n}$ :

\PP(S_n > 5n) = \PP\bigl(e^{tS_n} > e^{5nt}\bigr) \le \frac{\EE\bigl[e^{tS_n}\bigr]}{e^{5nt}}. \tag{8.16}

Why $e^{tx}$ ? I was confused why this particular function. We want to apply a function $T$ so that $\EE\bigl[T(\sum_i P_i)\bigr]$ splits into a product $\prod_i \EE[T(P_i)]$ by independence. That requires $T(x + y) = T(x)\,T(y)$ , which the exponential satisfies: $T(x + y) = e^{t(x+y)} = e^{tx}\, e^{ty} = T(x)\,T(y)$ . The parameter $t > 0$ is a scaling factor we are free to choose.

Since the $P_i$ are independent and identically distributed,

\EE\bigl[e^{tS_n}\bigr] = \EE\bigl[e^{tP_1 + tP_2 + \cdots + tP_n}\bigr] = \EE\bigl[e^{tP_1}\bigr]\,\EE\bigl[e^{tP_2}\bigr] \cdots \EE\bigl[e^{tP_n}\bigr] = \Bigl(\EE\bigl[e^{tP_1}\bigr]\Bigr)^n.

For $P \sim \Pois(\lambda)$ , using the Taylor series $e^{x} = \sum_{k \ge 0} \frac{x^k}{k!}$ ,

\EE\bigl[e^{tP}\bigr] = \sum_{k=0}^{\infty} e^{tk}\, \frac{e^{-\lambda}\lambda^k}{k!} = e^{-\lambda} \sum_{k=0}^{\infty} \frac{(e^{t}\lambda)^k}{k!} = e^{-\lambda}\, e^{e^t \lambda} = e^{\lambda(e^t - 1)}. \tag{8.18}

Substituting into (8.16),

\PP(S_n > 5n) \le \frac{\bigl(e^{\lambda(e^t - 1)}\bigr)^n}{e^{5nt}} = e^{\lambda n (e^t - 1) - 5nt} \overset{\lambda = 1}{=} e^{\,n\,(e^t - 1 - 5t)}.

Choosing $t$ . The bound holds for every $t > 0$ , so minimize the exponent $g(t) = e^t - 1 - 5t$ :

g'(t) = e^t - 5 = 0 \;\Longrightarrow\; t^\star = \ln 5, \qquad g''(t) = e^t > 0,

g(t^\star) = e^{\ln 5} - 1 - 5\ln 5 = 4 - 5\ln 5 \approx -4.05.

Therefore

\PP\Bigl(\sum_{i=1}^n P_i > 5n\Bigr) \le e^{-(5\ln 5 - 4)\,n} \approx e^{-4.05\,n}.

For $n = 10$ : Markov gives $0.2$ , while this bound gives $e^{-40.5} \approx 2.7 \times 10^{-18}$ .

Is the bound $\propto e^{-n}$ ? I was unsure about the rate. Any $t > 0$ with $g(t) < 0$ gives exponential decay $e^{n g(t)}$ ; the best rate is $5\ln 5 - 4 \approx 4.05$ , at $t^\star = \ln 5$ . This method (Markov applied to $e^{tX}$ , then optimizing over $t$ ) is called the Chernoff bound. The quantity $\EE[e^{tP}]$ is the MGF of $P$ , defined next.

Moment Generating Functions

Definition 8.2 (Moment generating function)

The moment generating function of a random variable $X$ is

M_X(s) \triangleq \EE\bigl[e^{sX}\bigr] = \begin{cases} \displaystyle \sum_x e^{sx}\, p_X(x) & \text{(discrete)}, \\[10pt] \displaystyle \int_{-\infty}^{\infty} e^{sx}\, f_X(x)\, dx & \text{(continuous)}, \end{cases}

for $s \in \RR$ (it may equal $\infty$ for some $s$ ).

MGFs are useful for:

tight tail bounds (Section 8.4);
computing all the moments $\EE[X^n]$ of a distribution;
sums of independent random variables: convolution becomes multiplication.

Theorem 8.3 (MGFs generate moments)

If $M_X(s) < \infty$ for all $s$ in an open interval around $0$ , then for every $n \ge 1$ ,

M_X^{(n)}(0) = \EE[X^n].

Proof.

Here $s$ is a constant inside the expectation and $X$ is the random part, so $\frac{d}{ds} e^{sX} = X e^{sX}$ . Differentiating $n$ times and moving the derivative inside the expectation (allowed because $M_X$ is finite near $0$ ),

\frac{d^n}{ds^n} M_X(s) = \frac{d^n}{ds^n} \EE\bigl[e^{sX}\bigr] = \EE\left[\frac{d^n}{ds^n} e^{sX}\right] = \EE\bigl[X^n e^{sX}\bigr].

At $s = 0$ , $e^{0 \cdot X} = e^0 = 1$ , so $M_X^{(n)}(0) = \EE\bigl[X^n e^{sX}\bigr]\big|_{s=0} = \EE[X^n]$ .

Example 8.4 (MGF of the exponential)

Let $X \sim \operatorname{Exponential}(\lambda)$ : $f_X(x) = \lambda e^{-\lambda x}$ for $x \in [0, \infty)$ . Then

M_X(s) = \int_0^{\infty} e^{sx}\, \lambda e^{-\lambda x}\, dx = \lambda \int_0^{\infty} e^{(s - \lambda)x}\, dx.

If $s < \lambda$ , the exponent $s - \lambda$ is negative and

M_X(s) = \lambda \left[\frac{1}{s - \lambda}\, e^{(s - \lambda)x}\right]_0^{\infty} = \lambda \left(0 - \frac{1}{s - \lambda}\right) = \frac{\lambda}{\lambda - s}.

If $s \ge \lambda$ , the integrand $e^{(s-\lambda)x} \ge 1$ for all $x \ge 0$ , so the integral diverges:

M_X(s) = \begin{cases} \dfrac{\lambda}{\lambda - s} & s < \lambda, \\[8pt] \infty & s \ge \lambda. \end{cases}

Moments. $M_X'(s) = \frac{\lambda}{(\lambda - s)^2}$ and $M_X''(s) = \frac{2\lambda}{(\lambda - s)^3}$ , so

\EE[X] = M_X'(0) = \frac{1}{\lambda}, \qquad \EE[X^2] = M_X''(0) = \frac{2}{\lambda^2}, \qquad \Var(X) = \frac{2}{\lambda^2} - \frac{1}{\lambda^2} = \frac{1}{\lambda^2}.

Theorem 8.4 (MGFs determine the distribution)

If $M_X(s) = M_Y(s) < \infty$ for all $s$ in some open interval $(-\varepsilon, \varepsilon)$ around $0$ , then $X$ and $Y$ have the same distribution.

So if we recognize an MGF, we know the distribution. (The proof is beyond the scope of this course.)

Not finished in lecture. We ran out of time after starting the list of MGF properties. The main properties are filled in below.

Theorem 8.5 (Properties of MGFs)

$M_X(0) = 1$ .
If $Y = aX + b$ , then $M_Y(s) = e^{sb}\, M_X(as)$ .
If $X$ and $Y$ are independent, then $M_{X+Y}(s) = M_X(s)\, M_Y(s)$ . More generally, for independent $X_1, \ldots, X_n$ , $M_{X_1 + \cdots + X_n}(s) = \prod_{i=1}^n M_{X_i}(s)$ , and $\bigl(M_{X_1}(s)\bigr)^n$ if they are also identically distributed.

Proof.

$M_X(0) = \EE\bigl[e^{0 \cdot X}\bigr] = \EE[1] = 1$ .
$M_Y(s) = \EE\bigl[e^{s(aX + b)}\bigr] = \EE\bigl[e^{sb}\, e^{(as)X}\bigr] = e^{sb}\, \EE\bigl[e^{(as)X}\bigr] = e^{sb}\, M_X(as)$ , since $e^{sb}$ is a constant.
$M_{X+Y}(s) = \EE\bigl[e^{s(X+Y)}\bigr] = \EE\bigl[e^{sX}\, e^{sY}\bigr] = \EE\bigl[e^{sX}\bigr]\,\EE\bigl[e^{sY}\bigr] = M_X(s)\, M_Y(s)$ , using independence of $e^{sX}$ and $e^{sY}$ . The $n$ -term version follows by induction.

Property 3 is the payoff: the density of $X + Y$ requires a convolution integral, but its MGF is just a product.

MGF Proofs for Sums of Independent Random Variables

Lemma 8.1 (MGF of the Gaussian)

If $X \sim \mathcal{N}(\mu, \sigma^2)$ , then for all $s \in \RR$ ,

M_X(s) = e^{\mu s + \sigma^2 s^2 / 2}.

Proof.

First let $Z \sim \mathcal{N}(0, 1)$ . Complete the square in the exponent, $z^2 - 2sz = (z - s)^2 - s^2$ :

\begin{aligned} M_Z(s) = \int_{-\infty}^{\infty} e^{sz}\, \frac{1}{\sqrt{2\pi}}\, e^{-z^2/2}\, dz &= \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, e^{-(z^2 - 2sz)/2}\, dz \\ &= \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, e^{-(z - s)^2/2 + s^2/2}\, dz \\ &= e^{s^2/2} \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, e^{-(z - s)^2/2}\, dz = e^{s^2/2} \cdot 1, \end{aligned}

since the last integrand is the $\mathcal{N}(s, 1)$ density. Now $X = \sigma Z + \mu$ (Lecture 7), so by Property 2 with $a = \sigma$ , $b = \mu$ ,

M_X(s) = e^{\mu s}\, M_Z(\sigma s) = e^{\mu s}\, e^{\sigma^2 s^2/2} = e^{\mu s + \sigma^2 s^2/2}.

Theorem 8.6 (Sum of independent Gaussians)

If $X \sim \mathcal{N}(\mu_X, \sigma_X^2)$ and $Y \sim \mathcal{N}(\mu_Y, \sigma_Y^2)$ are independent, then

X + Y \sim \mathcal{N}\bigl(\mu_X + \mu_Y,\; \sigma_X^2 + \sigma_Y^2\bigr).

Proof.

By Property 3 and the Gaussian MGF,

\begin{aligned} M_{X+Y}(s) = M_X(s)\, M_Y(s) &= e^{\mu_X s + \sigma_X^2 s^2/2} \cdot e^{\mu_Y s + \sigma_Y^2 s^2/2} \\ &= e^{(\mu_X + \mu_Y)s + (\sigma_X^2 + \sigma_Y^2)s^2/2}, \end{aligned}

which is the MGF of $\mathcal{N}(\mu_X + \mu_Y,\, \sigma_X^2 + \sigma_Y^2)$ , finite for all $s$ . Since MGFs determine the distribution, $X + Y \sim \mathcal{N}(\mu_X + \mu_Y,\, \sigma_X^2 + \sigma_Y^2)$ .

This completes the proof left as TBD in Lecture 7.

Corollary 8.1 (Sum of independent Poissons)

If $X \sim \Pois(\lambda)$ and $Y \sim \Pois(\mu)$ are independent, then $X + Y \sim \Pois(\lambda + \mu)$ .

Proof.

By (8.18), $M_X(s) = e^{\lambda(e^s - 1)}$ and $M_Y(s) = e^{\mu(e^s - 1)}$ , so

M_{X+Y}(s) = e^{\lambda(e^s - 1)}\, e^{\mu(e^s - 1)} = e^{(\lambda + \mu)(e^s - 1)},

the MGF of $\Pois(\lambda + \mu)$ . (Compare the convolution proof of Poisson merging in Lecture 4.)

sums-of-random-variables-and-moment-generating-functions.html

Lecture 8: Sums of Independent Random Variables and Moment Generating Functions

Discrete Convolution

Continuous Convolution

Example: Sum of Two Uniforms

Bounding Tail Probabilities

Moment Generating Functions

MGF Proofs for Sums of Independent Random Variables