05 Other density families

Sampling distributions

Sampling distributions are distributions of the functions of samples drawn from other distributions.

Standard normal square sum

Aka Chi square distribution with k degrees of freedom.

If $X_i\sim N(0,1)$, $\sum _{i=1}^k{X}_i^2\sim {\chi }_k^2$. This is same as the distribution of $\sum (\frac{Y_i-\mu }{\mu }{)}^2$.

Used in goodness of fit tests. \chk

Student’s t distribution with k degrees of freedom

$\frac{Z}{\sqrt{W/n}}$, with $Z\sim N(0,1),W\sim {\chi }_n^2,Z\perp W$.

F distribution

\(\frac{w_{1}/n_{1}}{w_{2}/ n_{2}} \distr F_{n_{1}, n_{2}} : w_{i} \distr \chi^{2}{n{i}}\).

Heavy tailed distributions

$l{t}_{x\to \mathrm{\infty }}\frac{Pr(X>x)}{e^{-ϵx}}=\mathrm{\infty }$. Eg: Power law distribution, cauchy distribution.

Power law distributions

$p(x)\propto x^{-g};p(x)=x^{-g}{Z}^{-1}$ for normalizing constant Z. $l{t}_{x\to 0}p(x)=\mathrm{\infty }$: so must have lower bound $x_{min}$. log p vs log x graph looks like a straight line.

Aka scale free distribution. The only \why distribution with the property: $\mathrm{\exists }g(b):p(bx)=g(b)p(x)$.

A subset of heavy-tailed distribution family.

Includes Zipf’s law distribution.

With exponential cutoff

$p(x)\propto x^{-\alpha }{e}^{-x\beta }$. log p vs log x graph looks like a straight line which suddenly bends: exponential term starts kicking in. Akin to gamma distribution.

Zipf’s law for resource usage

Frequency/ probability of usage of resources often follows Zipf’s law: Pr([res used]) $\propto f(resource{)}^{-k}$. Eg: words used in document.

Mixture distribution

Often, one models the pdf of $X$ as being a convex combination of multiple pdf’s.

Other pdf’s

Uniform and triangular distributions

Uniform distribution; used when not information is available except min, max. Triangular distribution is used when mode is also known.

Log normal distribution

Take $X\sim N(\mu ,{\sigma }^2)$. Then $Y=e^X$ has log normal distribution. Wide variety of shapes, heavy tailed.

Gumbel distribution

Used in worst case analysis. CDF: $G(x|\mu ,b)=e^{-e^{-\frac{x-\mu }{b}}}$, PDF: $g(x|\mu ,b)=\frac{e^{-\frac{x-\mu }{b}}}{b}{e}^{-e^{-\frac{x-\mu }{b}}}$.

Probability simplex coordinate powering

Aka Dirichlet distribution. This is the conjugate prior for multinomial distribution.

Support is $\{x\in R^k:\sum _i{x}_i=1,x_i>0\}$: or actually $\{x\in R^{k-1}:\sum _i{x}_i<1,x_i>0\}$. pdf is $p(x;a)\propto \prod _{i=1:k}{x}_i^{a_i-1}$ for parameters $a\ge 0$.

2-dim case

Aka beta(a,b) distribution. This takes up a wide variety of shapes: convex, concave, neither etc..

This is the conjugate prior for bernoulli/ binomial distribution - and a special case of Dirichlet distribution.

Pdf: $f(x)\propto x^{a-1}(1-x{)}^{b-1}$ for $x\in [0,1]$.

Wigner semicircle distribution

Supported on [-R, R], like a semicircle.