.

Bayesian Inference

In the classical approach, a probability is regarded as a long run frequency/propensity. We take random samples from a population of infinite size, and parameter $\theta$ is fixed.

In the Bayesian approach, a probability is a subjective degree of belief. Parameters themselves are regarded as random, and beliefs are updated based on data.

Ingredients

• Model for the data given some known parameters, $f_{\left.X\right|p}\left(\left.x\right|p\right)$.
• Prior distribution of parameters, $f_{p}\left(p\right)$.
• Bayes Theorem: $\underset{\text{posterior distribution}}{\underbrace{f_{\left.p\right|X}\left(\left.p\right|x\right)}}=\frac{f_{X,p}\left(X,p\right)}{f_{X}\left(x\right)}=\frac{\overset{\text{likelihood function}}{\overbrace{f_{\left.X\right|p}\left(\left.x\right|p\right)}}.\overset{\text{prior distribution}}{\overbrace{f_{p}\left(p\right)}}}{\int f_{\left.X\right|p}\left(\left.x\right|p\right)f_{p}\left(p\right)dp}$

In this framework, estimating a parameter means finding $f_{\left.p\right|X}$ for some data $x_{1}..x_{n}$. Unlike in the classical approach, parameters are now distributed according to function $f_{p}\left(\cdot\right)$, although we’ll keep writing them in lowercase. In actual estimation, this function may encompass the researcher’s past experience or data from previous experiments. In practice, the function is often chosen to be relatively uninformative.

The posterior distribution is the updated distribution of the parameters, conditional on data. We may use the distribution to generate point estimates (by taking expectations of the parameters, for example).