9 Bayesian Modelling

Author

Adrien Osakwe

In this section we will explore extensions of the Bayesian methods framework for other modelling:

regression
latent variable models
hierarchical models

9.1 Regression Models

We can consider an infinite sequence ${(X_{n}, Y_{n}), n = 1, 2, \dots}$ such that for any $n \geq 1$

$f_{X_{1}, . . ., X_{n}, Y_{1}, . . ., Y_{n}} (x_{1}, . . ., x_{n}, y_{1}, . . ., y_{n})$

can be factorized as

$f_{X_{1}, . . ., X_{n}} (x_{1}, . . ., x_{n}) f_{Y_{1}, . . ., Y_{n} | X_{1}, . . ., X_{n}} (y_{1}, . . ., y_{n} | x_{1}, . . ., x_{n})$

where each term has a deFinetti representation.

$f_{X_{1}, . . ., X_{n}} (x_{1}, . . ., x_{n}) = \int {\prod_{i = 1}^{n} f_{X} (x_{i}; ϕ)} π_{0} (d ϕ) f_{Y_{1}, . . ., Y_{n} | X_{1}, . . ., X_{n}} (y_{1}, . . ., y_{n} | x_{1}, . . ., x_{n}) = \int {\prod_{i = 1}^{n} f_{Y | X} (y_{i} | x_{i}; θ)} π_{0} (d θ)$

Given the above structure, inference for $(ϕ, θ)$ is required:

inference for $ϕ$ is done through the marginal model for the X variables
inference for $θ$ is done through the conditional model for Y given that X is observed.

For the latter, the fact that X is random is irrelevant as we have conditioned the model on observed values of X.

When considering the statistical behaviour of Bayesian (and frequentist) procedures, we need to remember that X and Y have a joint structure.

Prediction

$f_{Y_{n + 1} | X_{1 : n}, Y_{1 : n}} (y_{n + 1} | x_{1 : n}, y_{1 : n}) = \int f_{X_{n + 1}, Y_{n + 1} | X_{1 : n}, Y_{1 : n}} (x_{n + 1}, y_{n + 1} | x_{1 : n}, y_{1 : n}) d x_{n + 1} = \int f_{Y_{n + 1} | X_{1 : n}, X_{n + 1}, Y_{1 : n}} (y_{n + 1} | x_{1 : n}, x_{n + 1}, y_{1 : n}) f_{X_{n + 1} | X_{1 : n}, Y_{1 : n}} (x_{n + 1} | x_{1 : n}, y_{1 : n}) d x_{n + 1}$

9.2 Linear Regression

We can start with the following linear regression model

$Y_{i} = x_{i} β + ϵ_{i}$

where for $i = 1, \dots, n$

$Y_{i}$ is a scalar
$x_{i}$ is (1 x d)
$β$ is (d x 1)
$ϵ_{i} \sim N o r m a l (0, σ^{2})$ , independently.

With this structure, we can describe the model for the partially exchangeable random variables (error terms) $ϵ_{i} = Y_{i} - x_{i} β$ , conditional on $X_{i} = x_{i}$ . In this scenario, there may or may not be a need to model the distribution of $X_{i}$ . –> Note: figure out why.

We can look at the vector form of the linear regression model

$Y = X β + ϵ$

where the response variable and the error terms are (n x 1) vectors and the predictors are an (nxd) matrix.

We can then have a conditional model

$f_{Y_{1}, . . ., Y_{n} | X_{1}, . . ., X_{n}} (y_{1}, . . ., y_{n} | x_{1}, . . ., x_{n}; β, σ^{2}) \equiv N o r m a l_{n} (X β, σ^{2} I_{n})$

where $I_{n}$ is an identity matrix (nxn).

With this structure, we know the likelihood to be

$L_{n} (β, σ^{2}) = (\frac{1}{2 π σ^{2}})^{n / 2} e x p {\frac{1}{2 σ^{2}} (y - X β)^{T} (y - X β)}$

We can derive a joint conjugate prior

$π_{0} (β, σ^{2}) = π_{0} (σ^{2}) π_{0} (β | σ^{2})$

where

$π_{0} (σ^{2}) \equiv I n v G a m m a (a_{0} / 2, b_{0} / 2) π_{0} (β | σ^{2}) \equiv N o r m a l_{d} (m_{0}, σ^{2} M_{0})$

where $a_{0}, b_{0}, m_{0}, M_{0}$ are user-defined constant hyperparameters. The joint posterior can hence be approximated with

$π_{n} (β, σ^{2}) \propto L_{n} (β, σ^{2}) π_{0} (β, σ^{2}) π_{0} (σ^{2}) = \frac{(b_{0} / 2)^{a_{0} / 2}}{Γ (a_{0} / 2)} (\frac{1}{σ^{2}})^{a_{0} / 2 + 1} e x p {- \frac{b_{0}}{2 σ^{2}}} π_{0} (β | σ^{2}) = (\frac{1}{2 π σ^{2}})^{d / 2} \frac{1}{| M_{0} |^{0.5}} e x p {\frac{1}{2 σ^{2}} (β - m_{0})^{T} M_{0}^{- 1} (β - m_{0})}$

We can explore the exponents of the above posterior as a quadratic form.

The expression

$(y - X β)^{T} (y - X β) + (β - m_{0})^{T} M_{0}^{- 1} (β - m_{0})$

which equates to $(β - m_{n})^{T} M_{n}^{- 1} (β - m_{n}) + c_{n}$ where we need to determine the expressions for $m_{n}, M_{n}, c_{n}$ .

Quadratic term:

$β^{T} M_{n}^{- 1} β = β^{T} X^{T} X β + β^{T} M_{0}^{- 1} β$

and therefore

$M_{n}^{- 1} = X^{T} X + M_{0}^{- 1}$ –>> $M_{n} = (X^{T} X + M_{0}^{- 1})^{- 1}$
Linear term:

$\beta^TM_n^{-1}m_n = \beta^TX^Ty + \beta^TM_0^{-1}m_0 $

and therefore

$m_{n} = M_{n} (X^{T} y + M_{0}^{- 1} m_{0})$

$= (X^{T} X + M_{0}^{- 1})^{- 1} (X^{T} y + M_{0}^{- 1} m_{0})$
Constant term:

$m_{n}^{T} M_{n}^{- 1} m_{n} + c_{n} = y^{T} y + m_{0}^{T} M_{0}^{- 1} m_{0}$

and therefore

$c_{n} = y^{T} y + m_{0}^{T} M_{0}^{- 1} m_{0} - m_{n}^{T} M_{n}^{- 1} m_{n}$

Given us the joint posterior (under proportionality) as

$π_{n} (β, σ^{2}) = (\frac{1}{σ^{2}})^{\frac{(n + a_{0} + d)}{2} + 1} \exp {- \frac{(c_{n} + b_{0})}{2 σ^{2}}} \exp {\frac{1}{2 σ^{2}} (β - m_{n})^{T} M_{n}^{- 1} (β - m_{n})}$

Which tells us that the conditional posterior and marginalizing the joint posterior over $β$ give us

$π_{n} (β | σ^{2}) \equiv N o r m a l_{d} (m_{n}, σ^{2} M_{n}) π_{n} (σ^{2}) \equiv I n v G a m m a (a_{n} / 2, b_{n} / 2)$

where $a_{n} = a_{0} + n$ and $b_{n} = b_{0} + c_{n}$ .

Note: see slides 218-221 for the marginal $β$ posterior where knowledge of the Inverse Gamma pdf will reveal that $π_{n} (β)$ is a multivariate Student-t distribution.

Assigning prior ignorance to $β$ (setting $M_{0}^{- 1}$ to 0) will lead to results that equate to what we would get from the maximum likelihood approach.

$m_{n} \to (X^{T} X)^{- 1} X^{T} y M_{n} \to (X^{T} X)^{- 1}$

We can alternatively use a g-prior: with hyperparameter $λ > 0$ .

$M_{0} = λ^{- 1} (X^{T} X)^{- 1}$

and hence

$M_{n} = (1 + λ)^{- 1} (X^{T} X)^{- 1}$

If, for the g-prior we have

$m_{0} = 0_{d} / / M_{0} = λ I_{d}$

then we will have

$m_{n} = (X^{T} X + λ I_{d})^{- 1} X^{T} y M_{n} = (X^{T} X + λ I_{d})^{- 1}$

which gives us the procedure for ridge regression.

Jeffrey’s prior for linear regression (see slide 228 for derivation):

$π_{0} (β, σ^{2}) \propto (\frac{1}{σ^{2}})^{d / 2 + 1}$

9 Bayesian Modelling

9.1 Regression Models

9.2 Linear Regression

9.3 Non-linear Regression

9.3.1 Generalized Linear Models