gaussian-derived-distributions-and-order-statistics.html

Lecture 7: The Gaussian Distribution, Derived Distributions, and Order Statistics

EECS 126 (UC Berkeley) · Spring 2026 · February 10, 2026

Reading: Bertsekas & Tsitsiklis, §3.3, §4.1.

These are my personal notes from Professor Kannan Ramchandran’s EECS 126 (Probability and Random Processes) course at UC Berkeley.

Topics: the Gaussian distribution, change of variables (derived distributions), and order statistics.

The Gaussian Distribution

Definition 7.1 (Gaussian (normal) distribution)

X∼N(μ,σ2)X \sim \mathcal{N}(\mu, \sigma^2), with μ∈R\mu \in \RR and σ>0\sigma > 0, if

fX(x)=12π σ e−(x−μ)2/2σ2,x∈R.f_X(x) = \frac{1}{\sqrt{2\pi}\,\sigma}\, e^{-(x-\mu)^2/2\sigma^2}, \qquad x \in \RR.

The standard normal is Z∼N(0,1)Z \sim \mathcal{N}(0, 1):

fZ(z)=12π e−z2/2,z∈R.(7.2)f_Z(z) = \frac{1}{\sqrt{2\pi}}\, e^{-z^2/2}, \qquad z \in \RR. \tag{7.2}

The constant 12π\frac{1}{\sqrt{2\pi}} is the normalization factor that makes ∫−∞∞fZ(z) dz=1\int_{-\infty}^{\infty} f_Z(z)\,dz = 1.

Definition 7.2 (Standard normal CDF)
Φ(z)≜P(Z≤z)=∫−∞z12π e−u2/2 du.\Phi(z) \triangleq \PP(Z \le z) = \int_{-\infty}^{z} \frac{1}{\sqrt{2\pi}}\, e^{-u^2/2}\, du.

uu is a dummy variable of integration; zz appears only in the upper limit. Φ\Phi has no closed form, so its values are read from a table.

Theorem 7.1 (Key Gaussian integral)

For every α>0\alpha > 0,

∫−∞∞e−αx2/2 dx=2πα,\int_{-\infty}^{\infty} e^{-\alpha x^2/2}\, dx = \sqrt{\frac{2\pi}{\alpha}},

or equivalently

I(α)≜12π∫−∞∞e−αx2/2 dx=1α=α−1/2.(7.5)I(\alpha) \triangleq \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-\alpha x^2/2}\, dx = \frac{1}{\sqrt{\alpha}} = \alpha^{-1/2}. \tag{7.5}
Proof.

Substitute u=α xu = \sqrt{\alpha}\,x, so du=α dxdu = \sqrt{\alpha}\,dx and αx2=u2\alpha x^2 = u^2:

∫−∞∞e−αx2/2 dx=∫−∞∞e−u2/2 duα=1α∫−∞∞e−u2/2 du=1α⋅2π,\int_{-\infty}^{\infty} e^{-\alpha x^2/2}\, dx = \int_{-\infty}^{\infty} e^{-u^2/2}\, \frac{du}{\sqrt{\alpha}} = \frac{1}{\sqrt{\alpha}} \int_{-\infty}^{\infty} e^{-u^2/2}\, du = \frac{1}{\sqrt{\alpha}} \cdot \sqrt{2\pi},

where the last step uses ∫−∞∞12πe−u2/2 du=1\int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}} e^{-u^2/2}\,du = 1 from (7.2). Dividing by 2π\sqrt{2\pi} gives (7.5).

Mean and Variance of the Standard Normal

Theorem 7.2 (Standard normal moments)

If Z∼N(0,1)Z \sim \mathcal{N}(0,1), then E[Z]=0\EE[Z] = 0 and Var⁡(Z)=1\Var(Z) = 1.

Proof of 𝔼[Z] = 0. E[Z]=∫−∞∞z fZ(z) dz=∫−∞∞12π z e−z2/2 dz.\EE[Z] = \int_{-\infty}^{\infty} z\, f_Z(z)\, dz = \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, z\, e^{-z^2/2}\, dz.

The integrand g(z)=z e−z2/2g(z) = z\,e^{-z^2/2} is odd, g(−z)=−g(z)g(-z) = -g(z), so the two halves cancel:

∫0∞z e−z2/2 dz=[−e−z2/2]0∞=0−(−1)=1,∫−∞0z e−z2/2 dz=[−e−z2/2]−∞0=−1−0=−1,\begin{aligned} \int_{0}^{\infty} z\, e^{-z^2/2}\, dz &= \Bigl[-e^{-z^2/2}\Bigr]_0^{\infty} = 0 - (-1) = 1, \\ \int_{-\infty}^{0} z\, e^{-z^2/2}\, dz &= \Bigl[-e^{-z^2/2}\Bigr]_{-\infty}^{0} = -1 - 0 = -1, \end{aligned}

and therefore E[Z]=12π(1−1)=0\EE[Z] = \frac{1}{\sqrt{2\pi}}(1 - 1) = 0.

Proof of Var(Z) = 1.

Differentiate e−αx2/2e^{-\alpha x^2/2} with respect to α\alpha:

ddα(e−αx2/2)=−x22 e−αx2/2⟹x2e−αx2/2=−2 ddα(e−αx2/2).\frac{d}{d\alpha}\Bigl(e^{-\alpha x^2/2}\Bigr) = -\frac{x^2}{2}\, e^{-\alpha x^2/2} \quad\Longrightarrow\quad x^2 e^{-\alpha x^2/2} = -2\, \frac{d}{d\alpha}\Bigl(e^{-\alpha x^2/2}\Bigr).

Then

E[Z2]=∫−∞∞x2 12π e−x2/2 dx=[∫−∞∞12π x2e−αx2/2 dx]α=1=[∫−∞∞−22π ddα(e−αx2/2) dx]α=1=[−2 ddα(12π∫−∞∞e−αx2/2 dx)]α=1(Leibniz rule)=[−2 ddα I(α)]α=1=[−2 ddα α−1/2]α=1by (7.5)=[−2(−12 α−3/2)]α=1=[α−3/2]α=1=[1α3]α=1=1.\begin{aligned} \EE[Z^2] &= \int_{-\infty}^{\infty} x^2\, \frac{1}{\sqrt{2\pi}}\, e^{-x^2/2}\, dx \\ &= \left[\int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, x^2 e^{-\alpha x^2/2}\, dx\right]_{\alpha=1} \\ &= \left[\int_{-\infty}^{\infty} \frac{-2}{\sqrt{2\pi}}\, \frac{d}{d\alpha}\Bigl(e^{-\alpha x^2/2}\Bigr)\, dx\right]_{\alpha=1} \\ &= \left[-2\, \frac{d}{d\alpha}\left(\frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} e^{-\alpha x^2/2}\, dx\right)\right]_{\alpha=1} && \text{(Leibniz rule)} \\ &= \left[-2\, \frac{d}{d\alpha}\, I(\alpha)\right]_{\alpha=1} = \left[-2\, \frac{d}{d\alpha}\, \alpha^{-1/2}\right]_{\alpha=1} && \text{by (7.5)} \\ &= \left[-2 \left(-\tfrac12\, \alpha^{-3/2}\right)\right]_{\alpha=1} = \left[\alpha^{-3/2}\right]_{\alpha=1} = \left[\frac{1}{\sqrt{\alpha^3}}\right]_{\alpha=1} = 1. \end{aligned}

The Leibniz rule (differentiation under the integral sign) justifies moving ddα\frac{d}{d\alpha} outside the integral. Finally,

Var⁡(Z)=E[Z2]−(E[Z])2=1−02=1.\Var(Z) = \EE[Z^2] - (\EE[Z])^2 = 1 - 0^2 = 1.

Why only α=1\alpha = 1? I was confused why we set α=1\alpha = 1, and whether it could be something else. The identity I(α)=α−1/2I(\alpha) = \alpha^{-1/2} holds for every α>0\alpha > 0, so II is a function of α\alpha that we can differentiate, and differentiating in α\alpha is what produces the factor x2x^2. We set α=1\alpha = 1 only after differentiating, because the standard normal has e−x2/2e^{-x^2/2}, i.e. α=1\alpha = 1. Setting α=1\alpha = 1 first would leave the constant I(1)=1I(1) = 1, whose derivative is 00.

Cross-check by integration by parts. With u=xu = x and dv=12π xe−x2/2 dxdv = \frac{1}{\sqrt{2\pi}}\, x e^{-x^2/2}\,dx, so v=−12π e−x2/2v = -\frac{1}{\sqrt{2\pi}}\, e^{-x^2/2}:

E[Z2]=[−x2π e−x2/2]−∞∞+∫−∞∞12π e−x2/2 dx=0+1=1.\EE[Z^2] = \left[-\frac{x}{\sqrt{2\pi}}\, e^{-x^2/2}\right]_{-\infty}^{\infty} + \int_{-\infty}^{\infty} \frac{1}{\sqrt{2\pi}}\, e^{-x^2/2}\, dx = 0 + 1 = 1.

Linear Transformations of Gaussians

Theorem 7.3 (Shifting and scaling a standard normal)

If Z∼N(0,1)Z \sim \mathcal{N}(0,1), σ>0\sigma > 0, and X=σZ+μX = \sigma Z + \mu, then X∼N(μ,σ2)X \sim \mathcal{N}(\mu, \sigma^2).

Proof.

Start with the CDF of XX. Since σ>0\sigma > 0, dividing by σ\sigma preserves the inequality:

FX(x)=P(X≤x)=P(σZ+μ≤x)=P(Z≤x−μσ)=FZ(z),z=x−μσ.F_X(x) = \PP(X \le x) = \PP(\sigma Z + \mu \le x) = \PP\left(Z \le \frac{x - \mu}{\sigma}\right) = F_Z(z), \qquad z = \frac{x - \mu}{\sigma}.

Differentiate with the chain rule, using dzdx=ddx(xσ−μσ)=1σ\frac{dz}{dx} = \frac{d}{dx}\left(\frac{x}{\sigma} - \frac{\mu}{\sigma}\right) = \frac{1}{\sigma}:

fX(x)=ddxFX(x)=ddxFZ(z)=fZ(z) dzdx=12π e−z2/2⋅1σ=12π σ e−(x−μ)2/2σ2.\begin{aligned} f_X(x) = \frac{d}{dx} F_X(x) = \frac{d}{dx} F_Z(z) = f_Z(z)\, \frac{dz}{dx} &= \frac{1}{\sqrt{2\pi}}\, e^{-z^2/2} \cdot \frac{1}{\sigma} \\ &= \frac{1}{\sqrt{2\pi}\,\sigma}\, e^{-(x-\mu)^2/2\sigma^2}. \end{aligned}

Is fZ(z)⋅1σf_Z(z) \cdot \frac{1}{\sigma} already the answer? I was unsure whether the line fX(x)=12πe−z2/2⋅1σf_X(x) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2} \cdot \frac{1}{\sigma} was correct. It is: substituting z=x−μσz = \frac{x-\mu}{\sigma} gives exactly the N(μ,σ2)\mathcal{N}(\mu, \sigma^2) density. The factor 1σ\frac{1}{\sigma} in front comes from dzdx\frac{dz}{dx}.

Should μ\mu and σ\sigma have a subscript ZZ? I was also confused about this. No: ZZ always has mean 00 and variance 11. Here μ\mu and σ\sigma are the shift and scale applied to ZZ, and they become the mean and standard deviation of XX: E[X]=σ E[Z]+μ=μ\EE[X] = \sigma\,\EE[Z] + \mu = \mu and Var⁡(X)=σ2Var⁡(Z)=σ2\Var(X) = \sigma^2 \Var(Z) = \sigma^2. If anything, they are μX\mu_X and σX\sigma_X.

Corollary 7.1 (Standardization)

If X∼N(μ,σ2)X \sim \mathcal{N}(\mu, \sigma^2), then Z=X−μσ∼N(0,1)Z = \dfrac{X - \mu}{\sigma} \sim \mathcal{N}(0, 1), and

FX(x)=Φ ⁣(x−μσ).F_X(x) = \Phi\!\left(\frac{x - \mu}{\sigma}\right).

Symmetry of Φ and Standardization

Theorem 7.4 (Symmetry of the standard normal CDF)

For every y∈Ry \in \RR,

Φ(−y)=1−Φ(y).\Phi(-y) = 1 - \Phi(y).

So a table only needs Φ(y)\Phi(y) for y≥0y \ge 0.

Proof.

fZf_Z is even: fZ(−v)=fZ(v)f_Z(-v) = f_Z(v). Substitute u=−vu = -v, du=−dvdu = -dv:

Φ(−y)=∫−∞−yfZ(u) du=∫∞yfZ(−v) (−dv)=∫y∞fZ(v) dv=P(Z≥y)=1−P(Z<y)=1−Φ(y),\begin{aligned} \Phi(-y) = \int_{-\infty}^{-y} f_Z(u)\, du &= \int_{\infty}^{y} f_Z(-v)\,(-dv) = \int_{y}^{\infty} f_Z(v)\, dv \\ &= \PP(Z \ge y) = 1 - \PP(Z < y) = 1 - \Phi(y), \end{aligned}

using P(Z<y)=P(Z≤y)\PP(Z < y) = \PP(Z \le y) for a continuous random variable.

Standard normal density with equal shaded tails: the left tail below −y is Φ(−y) and the right tail above y is 1 − Φ(y)
Example 7.1 (Annual rainfall)

The annual rainfall in a city is X∼N(60,202)X \sim \mathcal{N}(60, 20^2), in inches. Find P(X>80)\PP(X > 80).

Standardize with Z=X−6020∼N(0,1)Z = \frac{X - 60}{20} \sim \mathcal{N}(0,1):

P(X>80)=P(X−6020>80−6020)=P(X−6020>1)=P(Z>1)=1−P(Z≤1)=1−Φ(1)=1−0.8413=0.1587.\begin{aligned} \PP(X > 80) &= \PP\left(\frac{X - 60}{20} > \frac{80 - 60}{20}\right) \\ &= \PP\left(\frac{X - 60}{20} > 1\right) = \PP(Z > 1) \\ &= 1 - \PP(Z \le 1) = 1 - \Phi(1) = 1 - 0.8413 = 0.1587. \end{aligned}
Standard normal density with the right tail above z = 1 shaded, labeled P(Z > 1)

Why 1−Φ(1)1 - \Phi(1)? I was confused why the answer is 1−Φ(1)1 - \Phi(1) rather than Φ(1)\Phi(1). Φ(1)\Phi(1) is the area to the left of 11; P(Z>1)\PP(Z > 1) is the area to the right, which is its complement.

Sums of Independent Gaussians

Theorem 7.5 (Sum of independent Gaussians)

If X∼N(μX,σX2)X \sim \mathcal{N}(\mu_X, \sigma_X^2) and Y∼N(μY,σY2)Y \sim \mathcal{N}(\mu_Y, \sigma_Y^2) are independent, then

X+Y∼N(μX+μY,  σX2+σY2).X + Y \sim \mathcal{N}\bigl(\mu_X + \mu_Y,\; \sigma_X^2 + \sigma_Y^2\bigr).

The parameters follow from Lecture 4: E[X+Y]=E[X]+E[Y]=μX+μY\EE[X+Y] = \EE[X] + \EE[Y] = \mu_X + \mu_Y, and by independence Var⁡(X+Y)=Var⁡(X)+Var⁡(Y)=σX2+σY2\Var(X+Y) = \Var(X) + \Var(Y) = \sigma_X^2 + \sigma_Y^2. The content of the theorem is that X+YX + Y is again Gaussian. This can be shown with a convolution, or in one line with moment generating functions.

Proof: MGF proof TBD in a later lecture.

Theorem 7.6 (Converse)

If XX and YY are independent and X+YX + Y is Gaussian, then XX and YY are each Gaussian (allowing a constant, i.e. variance 00).

This direction is much harder; its proof is beyond the scope of this course.

Derived Distributions

Given XX with known density fXf_X and Y=g(X)Y = g(X), find fYf_Y.

Theorem 7.7 (CDF method)
FY(y)=P(Y≤y)=P(g(X)≤y)=P(X∈{x:g(x)≤y})=∫{x : g(x)≤y}fX(x) dx,F_Y(y) = \PP(Y \le y) = \PP\bigl(g(X) \le y\bigr) = \PP\bigl(X \in \{x : g(x) \le y\}\bigr) = \int_{\{x \,:\, g(x) \le y\}} f_X(x)\, dx,

and then

fY(y)=dFY(y)dy.f_Y(y) = \frac{dF_Y(y)}{dy}.

If gg is strictly increasing, {x:g(x)≤y}={x≤g−1(y)}\{x : g(x) \le y\} = \{x \le g^{-1}(y)\}, so FY(y)=FX(g−1(y))F_Y(y) = F_X\bigl(g^{-1}(y)\bigr). If gg is strictly decreasing, the inequality flips: {x:g(x)≤y}={x≥g−1(y)}\{x : g(x) \le y\} = \{x \ge g^{-1}(y)\}, so FY(y)=1−FX(g−1(y))F_Y(y) = 1 - F_X\bigl(g^{-1}(y)\bigr).

Corollary 7.2 (Monotone change of variables)

If gg is strictly monotone and differentiable, then

fY(y)=fX(g−1(y))∣d g−1(y)dy∣.(7.21)f_Y(y) = f_X\bigl(g^{-1}(y)\bigr) \left| \frac{d\, g^{-1}(y)}{dy} \right|. \tag{7.21}
Proof.

Increasing gg: ddyFX(g−1(y))=fX(g−1(y))dg−1(y)dy\frac{d}{dy} F_X(g^{-1}(y)) = f_X(g^{-1}(y)) \frac{d g^{-1}(y)}{dy}, with dg−1(y)dy>0\frac{d g^{-1}(y)}{dy} > 0. Decreasing gg: ddy[1−FX(g−1(y))]=−fX(g−1(y))dg−1(y)dy\frac{d}{dy}\bigl[1 - F_X(g^{-1}(y))\bigr] = -f_X(g^{-1}(y)) \frac{d g^{-1}(y)}{dy}, with dg−1(y)dy<0\frac{d g^{-1}(y)}{dy} < 0. In both cases the result is (7.21).

For X=σZ+μX = \sigma Z + \mu: g−1(x)=x−μσg^{-1}(x) = \frac{x-\mu}{\sigma} and dg−1(x)dx=1σ\frac{d g^{-1}(x)}{dx} = \frac{1}{\sigma}, which is the factor 1σ\frac{1}{\sigma} found in Section 7.3.

Example 7.2 (Square root of a uniform)

Let X∼Uniform⁡[0,1]X \sim \operatorname{Uniform}[0,1] and Y=XY = \sqrt{X}. For 0≤y≤10 \le y \le 1, since x\sqrt{x} is increasing on [0,1][0,1] and both sides are nonnegative,

FY(y)=P(Y≤y)=P(X≤y)=P(X≤y2)=FX(y2)=∫0y21 dx=[x]0y2=y2−0=y2,F_Y(y) = \PP(Y \le y) = \PP(\sqrt{X} \le y) = \PP(X \le y^2) = F_X(y^2) = \int_0^{y^2} 1\, dx = \Bigl[x\Bigr]_0^{y^2} = y^2 - 0 = y^2,

with FY(y)=0F_Y(y) = 0 for y<0y < 0 and FY(y)=1F_Y(y) = 1 for y>1y > 1. Differentiating,

fY(y)=ddyFY(y)=ddy(y2)=2y,0≤y≤1.f_Y(y) = \frac{d}{dy} F_Y(y) = \frac{d}{dy}\bigl(y^2\bigr) = 2y, \qquad 0 \le y \le 1.

Check with (7.21): g−1(y)=y2g^{-1}(y) = y^2, dg−1(y)dy=2y\frac{d g^{-1}(y)}{dy} = 2y, fX(y2)=1f_X(y^2) = 1, so fY(y)=1⋅2y=2yf_Y(y) = 1 \cdot 2y = 2y.

Graph of y = √x on [0, 1]

Y=XY = \sqrt{X} has the same density 2y2y as the distance of a random dart from the center of the unit disk (Lecture 5).

Order Statistics

Let X1,X2,…,XnX_1, X_2, \ldots, X_n be i.i.d. samples with common density fXf_X and CDF FXF_X.

Definition 7.3 (Order statistics)
X(k)≜the k-th smallest of X1,X2,…,Xn,k=1,…,n.X^{(k)} \triangleq \text{the } k\text{-th smallest of } X_1, X_2, \ldots, X_n, \qquad k = 1, \ldots, n.

In particular X(1)=min⁡iXiX^{(1)} = \min_i X_i and X(n)=max⁡iXiX^{(n)} = \max_i X_i.

Example 7.3 (Ordering three samples)

n=3n = 3, Xi∼Uniform⁡(0,1)X_i \sim \operatorname{Uniform}(0,1), observed (X1,X2,X3)=(0.3, 0.8, 0.4)(X_1, X_2, X_3) = (0.3,\, 0.8,\, 0.4). Then

X(1)=0.3,X(2)=0.4,X(3)=0.8.X^{(1)} = 0.3, \qquad X^{(2)} = 0.4, \qquad X^{(3)} = 0.8.

Question: what is fX(k)(x)f_{X^{(k)}}(x), the density of X(k)X^{(k)}?

By the definition of a density,

P(X(k)∈(x,x+dx))≈fX(k)(x) dx.\PP\bigl(X^{(k)} \in (x, x + dx)\bigr) \approx f_{X^{(k)}}(x)\, dx.

For the kk-th smallest sample to lie in (x,x+dx)(x, x+dx), we need

  • k−1k - 1 samples in (−∞,x)(-\infty, x): probability [FX(x)]k−1\bigl[F_X(x)\bigr]^{k-1};
  • 11 sample in (x,x+dx)(x, x + dx): probability fX(x) dxf_X(x)\,dx;
  • n−kn - k samples in (x+dx,∞)(x + dx, \infty): probability [1−FX(x)]n−k\bigl[1 - F_X(x)\bigr]^{n-k},

where the probabilities multiply by independence.

Number line with k − 1 samples left of x, the k-th order statistic in the interval (x, x + dx), and n − k samples to the right

Count the ways to assign the nn samples to the three regions:

  • which sample lies in (x,x+dx)(x, x+dx): nn ways;
  • which k−1k-1 of the remaining n−1n-1 lie to the left: (n−1k−1)\binom{n-1}{k-1} ways;
  • the remaining n−kn-k lie to the right: 11 way.
n(n−1k−1)=n⋅(n−1)!(k−1)! ((n−1)−(k−1))!=n!(k−1)! (n−k)!.n \binom{n-1}{k-1} = n \cdot \frac{(n-1)!}{(k-1)!\,\bigl((n-1)-(k-1)\bigr)!} = \frac{n!}{(k-1)!\,(n-k)!}.

Why only 11 way for the third group? I was confused whether the right-hand group contributes ( ⋅ n−k)\binom{\,\cdot\,}{n-k} more choices. It does not: once the middle sample and the k−1k-1 left samples are chosen, the other n−kn-k samples are forced to the right. (Two or more samples landing in (x,x+dx)(x, x+dx) has probability of order (dx)2(dx)^2, which is negligible.)

Theorem 7.8 (Density of the kk-th order statistic)
fX(k)(x)=n!(k−1)! (n−k)! [FX(x)]k−1[1−FX(x)]n−kfX(x).(7.28)f_{X^{(k)}}(x) = \frac{n!}{(k-1)!\,(n-k)!}\, \bigl[F_X(x)\bigr]^{k-1} \bigl[1 - F_X(x)\bigr]^{n-k} f_X(x). \tag{7.28}
Proof.

Multiply the number of assignments by the probability of each:

fX(k)(x) dx=n!(k−1)! (n−k)! [FX(x)]k−1[1−FX(x)]n−kfX(x) dx,f_{X^{(k)}}(x)\, dx = \frac{n!}{(k-1)!\,(n-k)!}\, \bigl[F_X(x)\bigr]^{k-1} \bigl[1 - F_X(x)\bigr]^{n-k} f_X(x)\, dx,

and divide both sides by dxdx.

Why yy? I was confused why the formula was written in yy rather than xx. The argument is a dummy variable: fX(k)(y)f_{X^{(k)}}(y) and fX(k)(x)f_{X^{(k)}}(x) are the same function. What matters is that one variable is used throughout, and that FXF_X and fXf_X are the CDF and PDF of a single sample, not of X(k)X^{(k)}.

Check, k=nk = n (the maximum). P(X(n)≤x)=P(all Xi≤x)=[FX(x)]n\PP(X^{(n)} \le x) = \PP(\text{all } X_i \le x) = \bigl[F_X(x)\bigr]^n, and ddx[FX(x)]n=n[FX(x)]n−1fX(x)\frac{d}{dx}\bigl[F_X(x)\bigr]^n = n\bigl[F_X(x)\bigr]^{n-1} f_X(x), matching (7.28) with k=nk = n.

Corollary 7.3 (Uniform samples)

If Xi∼Uniform⁡[0,1]X_i \sim \operatorname{Uniform}[0,1], then fX(x)=1f_X(x) = 1 and FX(x)=xF_X(x) = x on [0,1][0,1], so

fX(k)(x)=n!(k−1)! (n−k)! xk−1(1−x)n−k,0≤x≤1.(7.30)f_{X^{(k)}}(x) = \frac{n!}{(k-1)!\,(n-k)!}\, x^{k-1} (1 - x)^{n-k}, \qquad 0 \le x \le 1. \tag{7.30}
On [0, 1], the Uniform density f_X(x) = 1 is a flat line and the CDF F_X(x) = x is the diagonal
Example 7.4 (Third smallest of four uniforms)

Xi∼Uniform⁡[0,1]X_i \sim \operatorname{Uniform}[0,1], n=4n = 4. Find P(X(3)<12)\PP\bigl(X^{(3)} < \tfrac12\bigr).

With n=4n = 4, k=3k = 3: n!(k−1)! (n−k)!=4!2! 1!=242=12\frac{n!}{(k-1)!\,(n-k)!} = \frac{4!}{2!\,1!} = \frac{24}{2} = 12. By (7.30),

P(X(3)<12)=FX(3)(12)=∫01/2fX(3)(x) dx=∫01/212 x2(1−x) dx=12∫01/2(x2−x3) dx=12[x33−x44]01/2=12(124−164)=12(8192−3192)=12⋅5192=60192=516.\begin{aligned} \PP\bigl(X^{(3)} < \tfrac12\bigr) = F_{X^{(3)}}\bigl(\tfrac12\bigr) &= \int_0^{1/2} f_{X^{(3)}}(x)\, dx = \int_0^{1/2} 12\, x^2 (1 - x)\, dx \\ &= 12 \int_0^{1/2} \bigl(x^2 - x^3\bigr)\, dx = 12 \left[\frac{x^3}{3} - \frac{x^4}{4}\right]_0^{1/2} \\ &= 12 \left(\frac{1}{24} - \frac{1}{64}\right) = 12 \left(\frac{8}{192} - \frac{3}{192}\right) = 12 \cdot \frac{5}{192} = \frac{60}{192} = \frac{5}{16}. \end{aligned}

Check: X(3)<12X^{(3)} < \tfrac12 exactly when at least 33 of the 44 samples are below 12\tfrac12. The number below 12\tfrac12 is Binomial⁡(4,12)\Bin(4, \tfrac12), so

P(X(3)<12)=(43)(12)4+(44)(12)4=416+116=516.\PP\bigl(X^{(3)} < \tfrac12\bigr) = \binom{4}{3}\Bigl(\tfrac12\Bigr)^4 + \binom{4}{4}\Bigl(\tfrac12\Bigr)^4 = \frac{4}{16} + \frac{1}{16} = \frac{5}{16}.
Example 7.5 (Expected minimum by symmetry)

Xi∼Uniform⁡[0,1]X_i \sim \operatorname{Uniform}[0,1], n=10n = 10. Find E[X(1)]\EE[X^{(1)}], the expected smallest sample.

The 1010 samples cut [0,1][0,1] into 1111 gaps:

Δ1=X(1)−0,Δi=X(i)−X(i−1)    (2≤i≤10),Δ11=1−X(10).\Delta_1 = X^{(1)} - 0, \qquad \Delta_i = X^{(i)} - X^{(i-1)} \;\; (2 \le i \le 10), \qquad \Delta_{11} = 1 - X^{(10)}.
Ten points on [0, 1] labeled X^(1) through X^(10), cutting the interval into gaps Δ1 through Δ11

The sum telescopes:

∑i=111Δi=(X(1)−0)+(X(2)−X(1))+⋯+(X(10)−X(9))+(1−X(10))=1.\sum_{i=1}^{11} \Delta_i = (X^{(1)} - 0) + (X^{(2)} - X^{(1)}) + \cdots + (X^{(10)} - X^{(9)}) + (1 - X^{(10)}) = 1.

By symmetry, all 1111 gaps have the same distribution, so E[Δi]=E[Δ1]\EE[\Delta_i] = \EE[\Delta_1] for every ii. Taking expectations,

E[∑i=111Δi]=∑i=111E[Δi]=11 E[Δ1]=E[1]=1⟹E[X(1)]=E[Δ1]=111.\EE\Bigl[\sum_{i=1}^{11} \Delta_i\Bigr] = \sum_{i=1}^{11} \EE[\Delta_i] = 11\, \EE[\Delta_1] = \EE[1] = 1 \quad\Longrightarrow\quad \EE[X^{(1)}] = \EE[\Delta_1] = \frac{1}{11}.

Do the gaps really sum to 11? I was unsure about this. They do: the gaps tile [0,1][0,1] exactly, which is the telescoping sum above.

Why are the gaps symmetric? Place 1111 independent uniform points on a circle of circumference 11 and cut the circle at one of them. The other 1010 points are i.i.d. uniform on the resulting interval [0,1][0,1], and the 1111 arcs between consecutive points are exactly the gaps Δ1,…,Δ11\Delta_1, \ldots, \Delta_{11}. On the circle no arc is special, so all gaps have the same distribution.

Cross-check with (7.30). With k=1k = 1: fX(1)(x)=10!0! 9!(1−x)9=10(1−x)9f_{X^{(1)}}(x) = \frac{10!}{0!\,9!}(1-x)^9 = 10(1-x)^9. Substituting u=1−xu = 1 - x,

E[X(1)]=∫01x⋅10(1−x)9 dx=∫01(1−u) 10 u9 du=10(110−111)=10⋅1110=111.\EE[X^{(1)}] = \int_0^1 x \cdot 10(1-x)^9\, dx = \int_0^1 (1 - u)\, 10\, u^9\, du = 10\left(\frac{1}{10} - \frac{1}{11}\right) = 10 \cdot \frac{1}{110} = \frac{1}{11}.

General case. X(k)=Δ1+⋯+ΔkX^{(k)} = \Delta_1 + \cdots + \Delta_k, so for nn uniform samples,

E[X(k)]=k E[Δ1]=kn+1.\EE[X^{(k)}] = k\, \EE[\Delta_1] = \frac{k}{n+1}.

Next lecture: moment generating functions (MGFs).

2324 words