Comparing Models using BIC and AIC

As George Box once said, “All models are wrong but some are useful.” There are no true models that perfectly reflect the data but AIC and BIC numbers can help us to make a judgment on which models better reflect the data than others.

AIC & BIC are both measures of the relative quality of a statistical model. AIC & BIC numbers do not provide a test of a model in the sense of testing a null hypothesis. They can tell nothing about the quality of the model in an absolute sense. They will not give any indication of poor fitting models.

What are AIC & BIC numbers?

Akaike Information Criterion (AIC) is an estimator of the relative expectation of Kullback-Leibler distance based on Fisher’s maximum likelihood. AIC estimates a constant plus the relative distance between the unknown true likelihood function of the data and the fitted likelihood function of the model. A smaller AIC number means a model is considered to be closer to the true model.

Bayesian Information Criterion (BIC) is an estimate of a function of the posterior probability of a model being true, under a certain Bayesian setup, so that a lower BIC means that a model is considered to be more likely to be the true model.

What’s the difference between AIC and BIC numbers?

When fitting models, it is possible to increase the likelihood by adding parameters, but doing so may result in overfitting. Both BIC and AIC resolve this problem by introducing a penalty term for the number of parameters in the model; the penalty term is larger in BIC than in AIC.

Which one to use?

AIC and BIC are both approximately correct according to a different goal and a different set of asymptotic assumptions. In order to compare AIC and BIC, we need to take a close look at the nature of the data generating model (such as having many tapering effects or not), whether the model set contains the generating model, and the sample sizes considered. Moreover, it also depends on weather we want to select the true model or select the K-L best approximating model. Thus, it is the unknown context and objectives of use (we want the “true” model or the “best” model) that is critical for deciding which measure is better.

In general, it might be best to use both AIC and BIC in model selection. AIC is better in situations when a false negative finding would be considered more misleading than a false positive, and BIC is better in situations where a false positive is as misleading as, or more misleading than, a false negative.

References:

Ask a Methodologist: AIC vs. BIC (2007, Spring). Retrieved from http://methodology.psu.edu/eresources/ask/sp07

Burnham, K. P., & Anderson, D. R. (2002). Model selection and multi-model inference: a practical information-theoretic approach. Springer.