6.1 Heuristics

Let \(l(\cdot)\) denote the loglikelihood, and let \(\tilde{\theta}\) denote the \(d\) dimensional parameter taking value in \(\Theta\subseteq \mathbb{R}^d\). We denote by \(\tilde{\theta}_0\) the true underlying parameter and by \(\hat{\theta}\) the MLE. Also, we assume that the MLE asymptotic normality holds, i.e. \[ \sqrt{n}\left(\hat{\theta}-\theta_0\right)\buildrel {\rm d} \over \longrightarrow N_d\left(\tilde{0},\left(I(\tilde{\theta}_0)\right)^{-1}\right) \] where \(I(\cdot)\) is the information matrix and is given by \[ I(\tilde{\theta})=\mathbb{E}{\left(\frac{\partial}{\partial \tilde{\theta}} f(X;\tilde{\theta})\frac{\partial}{\partial \tilde{\theta}} f(X;\tilde{\theta})^{\scriptscriptstyle\mathsf{T}}\right)}=-\mathbb{E}{\left(\frac{\partial^2}{\partial \tilde{\theta}^2} f(X;\tilde{\theta})\right)}. \] Now note that by using multivariate Taylor’s expansion we have, \[ \begin{aligned} l({\theta}_0) &\approx l(\hat{\theta}) + \dot{l}(\hat{\theta})^{\scriptscriptstyle\mathsf{T}} (\tilde{\theta}_0 - \hat{\theta}) + \frac{1}{2} (\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} \ddot{l}(\hat{\theta}) (\tilde{\theta}_0 - \hat{\theta})\\ &=l(\hat{\theta}) + \frac{1}{2} (\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} \ddot{l}(\hat{\theta}) (\tilde{\theta}_0 - \hat{\theta}) \end{aligned} \] In the above we denote the gradient and Hessian of \(l\) by \(\dot{l}\) and \(\ddot{l}\).
By using the consistency of the MLE, we have \(\hat{\theta}\buildrel {\rm d} \over \longrightarrow\tilde{\theta}_0\) this implies that \[ (\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} \ddot{l}(\hat{\theta}) (\tilde{\theta}_0 - \hat{\theta})=\sqrt{n}(\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} \left(\frac{1}{n}\ddot{l}(\hat{\theta})\right) \sqrt{n}(\tilde{\theta}_0 - \hat{\theta})\approx \sqrt{n}(\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} \left(\frac{1}{n}\ddot{l}(\tilde{\theta}_0)\right) \sqrt{n}(\tilde{\theta}_0 - \hat{\theta}). \] Now by using the LLN we have \(n^{-1}\ddot{l}(\tilde{\theta}_0)\buildrel {\rm d} \over \longrightarrow-I(\tilde{\theta}_0)\) \[ \sqrt{n}(\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} \left(\frac{1}{n}\ddot{l}(\tilde{\theta}_0)\right) \sqrt{n}(\tilde{\theta}_0 - \hat{\theta}) \approx -\sqrt{n}(\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} I(\tilde{\theta}_0) \sqrt{n}(\tilde{\theta}_0 - \hat{\theta}) \] Now note that if \[Y\sim N_d\left(\tilde{0},\Sigma\right)\] then
\[\Sigma^{-1/2}Y\sim N_d\left(\tilde{0},\Sigma^{-1/2}\Sigma\Sigma^{-1/2}\right)=N_d\left(\tilde{0},I_d\right).\] In particular, we have
\[ \Sigma^{-1/2}Y^{\scriptscriptstyle\mathsf{T}} \Sigma^{-1/2}Y \sim \chi^2_d.\] Combining this with the asymptotic normality of the MLE and the above observations, we have \[ (\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} \ddot{l}(\hat{\theta})(\tilde{\theta}_0 - \hat{\theta})\approx -\sqrt{n}(\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} I(\tilde{\theta}_0) \sqrt{n}(\tilde{\theta}_0 - \hat{\theta}) \buildrel {\rm d} \over \longrightarrow-\chi^2_d \] Hence, \[ \begin{aligned} l({\theta}_0) &\approx l(\hat{\theta}) + \frac{1}{2} (\tilde{\theta}_0 - \hat{\theta})^{\scriptscriptstyle\mathsf{T}} \ddot{l}(\hat{\theta}) (\tilde{\theta}_0 - \hat{\theta})\\ &\approx l(\hat{\theta}) - \frac{1}{2} \chi^2_d. \end{aligned} \] This suggests, that if we consider the region \(C_\alpha\subset\Theta\) defined by \[ C_\alpha:=\left\{\tilde{\theta} : \frac{1}{2} \chi^2_{d:\alpha} \geq l(\hat{\theta}) - l({\theta}) \right\}, \] where \(\chi^2_{d:\alpha}\) is the \(100(1-\alpha)\)-th percentile of \(\chi^2_{d}\), then the above implies that \[ {\rm Pr}\left( \tilde{\theta}_0 \in C_\alpha \right)\rightarrow 1-\alpha. \] In other words, \(C_\alpha\) is an asymptotic \(100(1-\alpha)%\) confidence region for \(\tilde{\theta}\).

6.2 Why use the above?

Note that one could construct joint confidence regions a few different ways. One is to use asymptotic normal confidence intervals for each coordinate of \(\tilde{\theta}\) and using the Bonferroni method construct a hyper-rectangle with asymptotic level at least \(1-\alpha\). An improvement over this rudimentary idea is to use the following (Wald’s Confidence Region) suggested by the asymptotic normality of the MLE: \[ \left\{\tilde{\theta}: -(\tilde{\theta}- \hat{\theta})^{\scriptscriptstyle\mathsf{T}} \ddot{l}(\hat{\theta}) (\tilde{\theta} - \hat{\theta}) \leq \chi^2_{d,\alpha} \right\}. \] But how do we choose between two confidence regions? The likelihood ratio based region is clearly transformation (of the parameter) invariant while the Wald’s region is not and in fact choosing the right parametrization (aka transformation) may help. The other is that these are asymptotics justified regions - aka asymptotic confidence regions. The level of \((1-\alpha)\) hence is nominal - for small sample size the exact confidence level may differ. The method which yields a confidence region whose exact confidence level is closer to the nominal even for small sample sizes is naturally the preferred one. We will see below that the likelihood ratio based region fares better; this is so as the Wald’s method forces the region to be an ellipsoid centered at the MLE - which is inherently symmetric and is unable to accommodate the small sample skewness.

6.3 One Dimensional Example - Exponential

Consider an iid sample of size \(n\) from the exponential distribution with mean \(\theta\). In this case the 95% Wald’s confidence region is given by \[ \bar{x} \pm \frac{1.96 \bar{x}}{\sqrt{n}}, \] using the fact that the MLE of \(\theta\) is the mean and the variance of the exponential is given by \(\theta^2\). Below, we check the exact confidence level of this nominal 95% interval, and we note that the choice of \(\theta_0\) is irrelevant as \(\theta\) is a scale parameter and the interval is scale equivariant. Also I simulated 1,000,000 intervals as the standard error then equals \[ \frac{\sqrt{0.95*.05}}{1000}\approx 0.02%; \] so even \(3\times\)s.d. is less that 0.1%.

Now we do the same for the likelihood ratio based confidence region. It is easy to see that this region is given by (see Example 13.14 in the text if need be)

\[ \left\{ \theta: \frac{\bar{x}}{\theta}+\log(\theta) \leq 1+\log(\bar{x})+\frac{1.92}{n}\right\}. \] Now I will use a similar simulation setup as above to compute the exact confidence level of the likelihood ratio based confidence region (which happens to be an interval too - why?)

To see the reason why the likelihood ratio confidence region does well, we for \(n=5\) construct the regions for \(\bar{x}\) equal to \(0.8,1\) and \(1.2\). Notice that while the Wald’s intervals are centered around the mean, the LR intervals are not.

6.4 Two Dimensional Example - Gamma

Consider an iid sample of size \(n\) from a Gamma distribution with shape parameter \(\alpha\) and scale parameter \(\theta\) - note that mean equals \(\alpha\theta\). We first recall some observations from the previous note, and derive the asymptotic variance-covariance of the MLE. Just to fix the parametrization, the density is given by

\[ f_{\alpha,\theta}(x)=\frac{1}{\Gamma(\alpha)\theta^\alpha} e^{-(x/\theta)} x^{\alpha-1 },\quad x\geq 0; \] the parameter space is given by \((0,\infty)^2\).