Skip to main content

Model Selection

Week 5. Which predictors belong in the model? Adding predictors always improves the fit to your own data and often makes predictions for new data worse. This week: why that happens, how AIC (and AICc and BIC) scores a model, and how to choose between models in a way that matches what the model is for: testing ideas you had before you saw the data, exploring a big data set for new hypotheses, or predicting.

Problem Set 5: Model Selection — due Monday, November 2, 1:10 PM. Submit on Canvas.

Answer key: Problem Set 5 key

Data: cover_crop.csv (cover-crop biomass on 50 farms with their weather, soil and location, simulated; the Week 2 live-coding data, back because we know the truth), pistacia_niche.csv (6,299 records of two wild pistachios, Pistacia lentiscus and P. terebinthus, with the 19 WorldClim climate variables and a sampling-effort column at each record; occurrence records from GBIF.org, download 0000438-260921141020460, 21 September 2026, thinned to about one record per species per climate pixel; climate from WorldClim 2.1), and for Problem Set 5: wild_wheat.csv (flowering time of 54 wild einkorn wheat accessions with the 19 WorldClim variables of their home sites; the Problem Set 2 data)

This week — Model selection

One deck across both lectures, in six sections. First lecture: Too many models (32,768 possible models from 15 predictors; R² always rises, error on new farms does not; overfitting; three goals: explore, test ideas, predict), Scoring models (likelihood, AIC by hand, ΔAIC, Akaike weights, AICc for small samples, BIC and when to use it), and Testing ideas (write the candidate models first, rank them with model.sel(), nested vs non-nested models, terms that stay in every model, and the two traps: a different response, or different rows). Second lecture: Exploring (a real data set with 19 tangled climate variables: an exact linear combination, why pairwise correlations are not enough, the VIF loop, a control variable kept with fixed =, dredge(), AICc vs BIC at large n, importance, model averaging), Traps (stepwise selection is unstable; selection makes pure noise look significant; confirm on data you did not use to choose; predictors are not causes), and Predicting (held-out data, AIC as an approximation to leave-one-out cross-validation, and the bridge to regularization next week).

  • Slides (PDF) — one deck for the week (the live-poll links are clickable)
  • Live coding in class — the short scripts we open in RStudio and run together, line by line: Week5_live_1_aic.R (overfitting, AIC by hand, AICc and BIC, six ideas compared with model.sel(), the two traps) and Week5_live_2_explore.R (the Pistacia data: a first model, collinearity and VIF, dredge(), importance and model averaging; forward and backward stepwise; pure noise; AIC vs leave-one-out). Plus Week5_noise_experiment.R: the pure-noise experiment on its own, to run on your laptop.
  • The week’s code, with output — one page for the whole week, in the same order as the slides, with the code behind every figure and number in the lecture and the output under each block. Every block is labelled Core (the analysis you would run on your own data), Handy (useful, not essential), or Demo (simulations and by-hand sums for the lecture; folded). Start with the Core blocks for the problem set.
  • Week5_Model_Selection.R — the same code as a plain script, each block tagged [CORE], [HANDY], or [DEMO]. It reads the data straight from the course website, so there is no working directory to set.

Fri Oct 30 — No class (instructor travel)

Learning objectives

  • Explain why R² always rises as predictors are added, and why prediction error on new data does not
  • Say which of three goals a model serves (exploration, inference, prediction) and pick a selection approach to match
  • Compute AIC from logLik() by hand, and use ΔAIC, Akaike weights, AICc and BIC
  • Write a small set of candidate models from hypotheses, compare them with model.sel(), and report the table
  • Recognize when AIC comparisons are invalid: a different response, or different rows
  • Handle collinearity before selection (exact combinations, the VIF loop), run dredge() with fixed control terms, and read importance and model-averaged estimates
  • Explain why stepwise selection and p-values after selection mislead, and confirm exploratory findings on held-out data

Also useful

  • Packages this week: MuMIn (for model.sel(), dredge(), AICc(), sw(), model.avg()), car (for vif()), MASS (for stepAIC()), ggplot2
  • Reading: Tredennick, Hooker, Ellner & Adler (2021), A practical guide to selecting models for exploration, inference, and prediction in ecology, Ecology 102(6): e03336
  • Textbook: ISLR (2nd edition) chapter 6, Linear Model Selection and Regularization: section 6.1 (subset selection, including Cp, AIC, BIC and adjusted R²); sections 6.2 onward are next week