Lecture 4: Variance, Covariance, and Common Discrete Distributions

No independence was used. The derivation above holds for any $X$ and $Y$. Independence only enters at the very end, where it makes the cross term vanish (Corollary 4.1 below).

Interpreting the sign. $\Cov(X,Y) > 0$ means $X$ and $Y$ tend to be above their means together (and below their means together). $\Cov(X,Y) < 0$ means when one is above its mean, the other tends to be below. For a sum, positive covariance inflates the spread and negative covariance dampens it.

Uncorrelated does not imply independent. Covariance only detects linear dependence. For example, let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$. Then $\EE[X] = 0$ and $\EE[XY] = \EE[X^3] = \tfrac13(-1) + \tfrac13(0) + \tfrac13(1) = 0$, so $\Cov(X, Y) = 0 - 0 \cdot \EE[Y] = 0$. But $Y$ is completely determined by $X$: for instance $\PP(Y = 0 \mid X = 0) = 1 \ne \tfrac13 = \PP(Y = 0)$. This was a question Ramchandran asked us at end of previous lecture.

Ramchandran’s vector analogy. Think of random variables as vectors, with “dot product” $\langle U, V\rangle = \EE[UV]$ and “length” $\sqrt{\EE[U^2]}$. For the centered variables $U = X - \EE[X]$ and $V = Y - \EE[Y]$:

• the dot product is $\EE[UV] = \Cov(X, Y)$;
• the lengths are $\sqrt{\EE[U^2]} = \sqrt{\Var(X)}$ and $\sqrt{\EE[V^2]} = \sqrt{\Var(Y)}$.

So $\rho$ plays the role of $\cos\theta = \dfrac{u \cdot v}{|u|\,|v|}$, the cosine of the angle between two vectors. For ordinary vectors, the Cauchy–Schwarz inequality $|u \cdot v| \le |u|\,|v|$ forces $-1 \le \cos\theta \le 1$. The same inequality, proved for random variables below, forces $-1 \le \rho \le 1$.

Extreme and zero values. $|\rho| = 1$ exactly when equality holds in Cauchy–Schwarz, which happens when $U - t^\star V = 0$ with probability $1$, i.e. when $Y$ is an exact linear function $Y = aX + b$ of $X$ (with $\rho = +1$ if $a > 0$ and $\rho = -1$ if $a < 0$). At the other end, $\rho = 0$ means uncorrelated, which, as we just saw, is weaker than independent.

Applying direct sum gives us the same result: Using the identity $x\binom{n}{x} = n\binom{n-1}{x-1}$ for $x \ge 1$ (the $x = 0$ term is zero),

where we substituted $j = x - 1$ and used the binomial theorem. The indicator method reaches the same answer with far less algebra.

Reading the PMF. Here $k$ marks the trial on which the first Success occurs ($k = S$). So $p_X(k)$ is the probability that the first success happens on trial $k$: the first $k - 1$ trials are failures (probability $(1-p)^{k-1}$) and trial $k$ is a success (probability $p$). Note that $p_X(k)$ is not the probability of $k$ successes; a geometric random variable always records exactly one success, and $k$ counts how long we waited for it.

Exam note: tail sum formula will be on exam. For a random variable $X$ taking values in $\{0, 1, 2, \ldots\}$,

It expresses the mean as the sum of the tail probabilities. It follows from writing $X = \sum_{k=1}^{\infty} \mathbf{1}\{X \ge k\}$ and taking expectations of both sides.

The direct sum, for comparison. Differentiating the geometric series $\sum_{k\ge0} x^k = \frac{1}{1-x}$ term by term (valid for $|x|<1$) gives $\sum_{k\ge1} k x^{k-1} = \frac{1}{(1-x)^2}$. With $x = 1-p$,

Mean equals variance (Poisson distribution). The Poisson is the only one of our four distributions whose mean and variance are equal. This is consistent with the Poisson being the limit of $\Bin(n, \lambda/n)$: the binomial variance is $n \cdot \frac{\lambda}{n}\bigl(1 - \frac{\lambda}{n}\bigr) = \lambda\bigl(1 - \frac{\lambda}{n}\bigr) \to \lambda$ as $n \to \infty$.

Numbers and intuition. For $N = 50$, $H_{50} \approx 4.499$, so $\EE[T] \approx 225$ boxes, about $4.5\times$ the number of coupon types. Most of the cost comes at the end: the final coupon alone takes $\EE[T_N] = N$ boxes on average, while the entire first half of the collection takes fewer than $N$ boxes in total ($\sum_{i \le N/2} \frac{N}{N-i+1} \le \frac{N}{2}\cdot 2 = N$). This $N\ln N$ behavior shows up whenever we need to “hit every bin” by random sampling: hashing, load balancing, and covering every node in a network.

Two remarks on memorylessness.

1. The remaining wait is again geometric. Given $X > s$, the number of additional trials $X - s$ has the same distribution as $X$: for $k \ge 1$, $$\PP(X - s = k \mid X > s) = \frac{\PP(X = s + k)}{\PP(X > s)} = \frac{(1-p)^{s+k-1}\,p}{(1-p)^{s}} = (1-p)^{k-1}\,p.$$ This is the form we use in the derivations below.
2. Uniqueness. The geometric is the only memoryless distribution on $\{1, 2, 3, \ldots\}$. Its continuous analog, the exponential distribution, will be the only memoryless continuous distribution.

Two checks on this argument.

1. Finiteness. Subtracting $\EE[X]$ from both sides is only legitimate because $\EE[X] < \infty$, which we already know from the tail sum. (If $\EE[X] = \infty$, the equation $\infty = \infty + 1 - \infty$ says nothing.)
2. Sanity check the answer. Any correct formula must satisfy $\EE[X] \ge 1$ (you need at least one trial) and $\EE[X] \to \infty$ as $p \to 0$. A sign slip in the algebra can produce $\EE[X] = \frac{1}{1+p}$, which fails both tests, since it is less than $1$ for every $p > 0$.