Beta-binomial Distribution - Further Bayesian Considerations

Further Bayesian Considerations

It is convenient to reparameterize the distributions so that the expected mean of the prior is a single parameter: Let

$\begin{align} \pi(\theta|\mu,M) & = \operatorname{Beta}(\mu,M) \\ & = \frac{\Gamma(M)}{\Gamma(\mu M)\Gamma(M(1-\mu))} \theta^{M\mu-1}(1-\theta)^{M(1-\mu)-1} \end{align}$

where

$\begin{align} \mu &= \frac{\alpha}{\alpha+\beta} \\ M &= \alpha+\beta \end{align}$

so that

$\begin{align} \operatorname{E}(\theta|\mu,M) & = \mu \\ \operatorname{Var}(\theta|\mu,M) & = \frac{\mu(1-\mu)}{M+1}. \end{align}$

The posterior distribution ρ(θ|k) is also a beta distribution:

$\begin{align} \rho(\theta|k) & \propto \ell(k|\theta)\pi(\theta|\mu,M) \\ & = \operatorname{Beta}(k+M \mu, n-k+M(1- \mu) ) \\ & = \frac{\Gamma(M)} {\Gamma(M\mu)\Gamma(M(1-\mu))} {n\choose k}\theta^{k+M\mu-1}(1-\theta)^{n-k+M(1-\mu)-1} \end{align}$

And

$\operatorname{E}(\theta|k) = \frac{k+M \mu}{n+M}.$

while the marginal distribution m(k|μ, M) is given by

$\begin{align} m(k|\mu,M) & = \int_0^1 l(k|\theta)\pi(\theta|\mu, M) \, d\theta \\ & = \frac{\Gamma(M)} {\Gamma(M\mu)\Gamma(M(1-\mu))} {n\choose k} \int_{0}^{1} \theta^{k+M\mu-1}(1-\theta)^{n-k+M(1-\mu)-1} d\theta \\ & = \frac{\Gamma(M)}{\Gamma(M\mu)\Gamma(M(1-\mu))} {n\choose k} \frac{\Gamma(k+M\mu)\Gamma(n-k+M(1-\mu))}{\Gamma(n+M)}. \end{align}$

Because the marginal is a complex, non-linear function of Gamma and Digamma functions, it is quite difficult to obtain a marginal maximum likelihood estimate (MMLE) for the mean and variance. Instead, we use the method of iterated expectations to find the expected value of the marginal moments.

Let us write our model as a two-stage compound sampling model. Let k_i be the number of success out of n_i trials for event i:

$\begin{align} k_i & \sim \operatorname{Bin}(n_i, \theta_i) \\ \theta_i & \sim \operatorname{Beta}(\mu,M),\ \mathrm{i.i.d.} \end{align}$

We can find iterated moment estimates for the mean and variance using the moments for the distributions in the two-stage model:

$\begin{align} \operatorname{var}\left(\frac{k}{n}\right) & = \operatorname{E}\left + \operatorname{var}\left \\ & = \operatorname{E}\left + \operatorname{var}\left(\theta|\mu,M\right) \\ & = \frac{1}{n}\left(\mu(1-\mu)\right) + \frac{n_{i}-1}{n_{i}}\frac{(\mu(1-\mu))}{M+1} \\ & = \frac{\mu(1-\mu)}{n}\left(1+\frac{n-1}{M+1}\right). \end{align}$

(Here we have used the law of total expectation and the law of total variance.)

We want point estimates for and . The estimated mean is calculated from the sample

The estimate of the hyperparameter M is obtained using the moment estimates for the variance of the two-stage model:

$s^2 = \frac{1}{N} \sum_{i=1}^N \operatorname{var}\left(\frac{k_{i}}{n_{i}} \right) = \frac{1}{N} \sum_{i=1}^N \frac{\hat{\mu}(1-\hat{\mu})}{n_i} \left$

Solving:

where

$s^2 = \frac{N \sum_{i=1}^N n_i (\hat{\theta_i} - \hat{\mu})^2 } {(N-1)\sum_{i=1}^N n_i }.$

Since we now have parameter point estimates, and, for the underlying distribution, we would like to find a point estimate for the probability of success for event i. This is the weighted average of the event estimate and . Given our point estimates for the prior, we may now plug in these values to find a point estimate for the posterior

Read more about this topic: Beta-binomial Distribution