2 Feb, 2016

Bayesian Interval Estimation

Exam

  • one note sheet
  • up to jeffreys prior

Bayesian Interval Estimation

A region $C \subset \Theta$ is a $100(1-\alpha)\%$ credibility if $P(\theta \in C | x) \ge (1-\alpha)$. $1-\alpha$ is the credibility level.

Under a jeffreys prior, the posterior is invariant to one-to-one transformations.

HPD
For a unimodal distribution, $\int_{c_1}^{c_2}~p(\theta | x) ~d\theta = (1-\alpha)$ and $c_2-c_1$ is the smallest possible length. For multidimensions distributions, the HPD may be a union of several regions. The heights will be the same.
  • Difficult to compute in practice (in multivariate space)
  • Not invariant under one-to-one transformations

Bayesian Asymptotics (Chapter 4, Berger; Section 4.7, Robert)

Result: $x_i \overset{iid}{\sim} p(x_i | \theta), \theta = (\theta_1,…,\theta_n)’$. For $\pi(\theta)$ the prior and $p(\theta | x)$ the posterior, and the prior and likelihood are twice differentiable around $\hat\theta$ the mle. As $n \rightarrow \infty$, the posterior can be approximated in 4 ways. These are also considered Laplace Approximations.

  1. $p(\theta | x) \approx N(\mu^\pi(x),v^\pi(x))$ where $(\mu^\pi(x), v^\pi(x))$ are the posterior mean and covariance.

  2. $p(\theta | x) \approx N_p(\hat\theta^\pi, [I^\pi(x)]^{-1})$, where $\hat\theta^\pi$ is the posterior mode and $[I^\pi(x)]_{ij} = -\paren{\frac{\partial^2}{\partial\theta_i \partial\theta_j} log[p(x | \theta)\pi(\theta)]}_{\theta=\hat\theta^\pi}$

  3. $p(\theta | x) \approx N_p(\hat\theta,(\hat{I}(x))^{-1})$, where $\hat I(x)$ is the observed Fishers info $[\hat I^\pi(x)]_{ij} = -\paren{\frac{\partial^2}{\partial\theta_i \partial\theta_j} log[p(x | \theta)]}_{\theta=\hat\theta} $

  4. $p(\theta | x) \approx N_p(\hat\theta, (I(\hat\theta))^{-1})$ with $I(\theta)$ the expected Fisher information matrix $[I(x)]_{ij} = -E\paren{\frac{\partial^2}{\partial\theta_i \partial\theta_j} log[p(x | \theta)]}$

Here is a helpful site for implementing Laplace approximations for Bayesian statistics.

Derivation for Laplace Approximation

Can we approximate $E(g(\theta) | x) = \frac{\int~g(\theta)p(x|\theta)\pi(\theta)~d\theta}{\int~p(x|\theta)\pi(\theta)~d\theta}$?

But let us first review laplace approximations. Let $I(n) = \displaystyle\int e^{-nh(\theta)} d\theta$

Let $\hat{\theta}$ be $\theta: h’(\theta)=0$. Then, $\hat{h}(\theta) \approx h(\theta) = \sum_{i=0}^\infty \frac{h^{(i)}(\hat\theta)}{i!}(\theta-\hat\theta)^i$. We take $\hat h$ to be the second order Taylor series expansion.

So, $I(n) \approx \int e^{-n\hat h}d\theta = \sqrt{2\pi\p{\frac{\sigma^2}{n}}} e^{-nh(\hat\theta)}$, where $\sigma^2 = \frac{1}{h’’(\hat\theta)}$.

In the multivariate case, $I(n) \approx (2\pi n)^{-p/2} \abs{\Sigma}^{1/2} \exp\p{-nh(\hat\theta)}$, where $\Sigma= \p{\frac{\partial^2 h(\theta)}{\partial\theta^2}\v_{\theta=\hat\theta}}^{-1}$.

Thus, we can use this approximation to get the results above.