2 Mean and variance

Mean: estimation

Consistency

Aka Law of large numbers

Let $\left\{X_i\right\}$ iid. \(\hat{X}{n} = n^{-1}\sum^{n} X{i}\). As $var[\widehat{X_n}]={\sigma }^2/n\to 0$ as $n\to \mathrm{\infty }$, Weak law: ${\hat{X}}_n$ is a consistent estimator of $\mu$.

Normalness of estimator distribution

Aka Central limit theorem (CLT)

Take estimator $U_n=\frac{\overline{X}-\mu }{\frac{\sigma }{\sqrt{n}}}$. $l{t}_{n\to \mathrm{\infty }}Pr(U_n\le u)={\int }_{-\mathrm{\infty }}^u\frac{1}{\sqrt{2\pi }}{e}^{-t^2/2}dt$: so approaches CDF of N(0,1): See convergence of moment generating function (MGF) below. So, as n increases, $var[\overline{X}]$ becomes smaller: visualize pdfs of \(X, \bar{X}{30}, \bar{X}{50}\); see how curve becomes more normal and gets thinner and taller. Generally, can use CLT when $n>30$.

Proof showing convergence to Normal MGF

Theorem: MGF \( M_{U_{n}(t) \to \) MGF of N(0, 1)}

Proof: iid $\left\{X_i\right\}$. $m_{U_n}(t)=E[e^{\frac{t(\sum X_i-n\mu )}{\sqrt{n}\sigma }}]=\prod E[e^{\frac{t}{\sqrt{n}}}(\frac{X_i-\mu )}{\sigma }]=m_Z(t/\sqrt{n}{)}^n$: implicitly defining Z with $E[Z]=0,var[Z]=E[Z^2]=1$.

But, by Taylor, $$m_Z(t/\sqrt{n})=m_Z(0)+m_Z^{\prime }(0)(t/\sqrt{n})+m_Z{\prime }^{\prime }(h)(t/\sqrt{n}{)}^2(1/2!)=1+E[Z]t+m^{}{\prime }^{\prime }(h)(\frac{t^2}{2n})$$ for some $h\in (0,t/\sqrt{n})$; so $m_Z(t/\sqrt{n})=1+m^{}{\prime }^{\prime }(h)(\frac{t^2}{2n})\to 1+\frac{t^2}{2n}$ as $n\to \mathrm{\infty }$. So, $m_{U_n}(t)\to (1+\frac{t^2}{2n}{)}^n\to e^{t^2/2}$, MGF of N(0, 1).

Normal distr: Pivotal quantity to estimate mean

Student’s t distribution is used to estimate $\mu$ when distribution is assumed to be Normal, n is small and $\sigma$ is unknown. Tables only go up to n = 30 or 40. If $\sigma$ were known, would use normal distribution, or if $n>30$ would estimate $\sigma$ and use normal distribution tables.

As $(n-1)\frac{S^2}{{\sigma }^2}\sim {\chi }_{n-1}^2$ ,

$\sqrt{n}\frac{\overline{X}-\mu }{S}\sim t_{n-1}$.

Goodness of empirical estimate

Can apply Chernoff bounds and Azuma Hoeffding inequality etc.. to judge goodness of empirical estimate.

For Binary valued random variables: A/B testing confidence interval, precision calculator here.

Variance estimation

The biased and unbiased estimators

$S^2=n^{-1}\sum (X_i-\overline{X}{)}^2$ biased: $B[S^2]=n^{-1}E(\sum X_i^2-2\overline{X}\sum X_i+n{\overline{X}}^2)-{\sigma }^2=n^{-1}(nE[X^2]-2E[n{\overline{X}}^2]+nE[{\overline{X}}^2])-{\sigma }^2=n^{-1}(n{\sigma }^2+n{\mu }^2-nvar[\overline{X}]+n{\mu }^2)-{\sigma }^2\to n^{-1}(n-1){\sigma }^2-{\sigma }^2$ from central limit thm. So, defined as $S^2=(n-1{)}^{-1}\sum (X_i-\overline{X}{)}^2$ to get unbiased estimator. Difference small as $n\to \mathrm{\infty }$.

Normal distr: Pivotal quantity to estimate variance

$N(\mu ,{\sigma }^2)$ assumed. If $S^2=\frac{\sum (X_i-\overline{X}{)}^2}{n-1}$, then $(n-1)\frac{S^2}{{\sigma }^2}\sim {\chi }_{n-1}^2$ \why.

So, can use this as pivotal quantity.

Sequential data Sample statistics

k-step Moving averages

Suppose that $ran(X_i)\in R$, and that the sample size is $n$.

Simple moving average

This is simply the mean of the last $k$ $X_i$.

Exponential Weighed

Here, one uses an exponentially decreasing weight (with decreasing $i$) while taking a weighted average of $k$ $X_i$.

Applications

This is useful while predicting stock prices, for example.