10. Data Summary for Multiple Regression

metricsAI: An Introduction to Econometrics with Python and AI in the Cloud

Carlos Mendez

Chapter 10 Visual Summary

This notebook provides an interactive introduction to multiple regression analysis. You’ll learn how to work with multiple explanatory variables, interpret partial effects, assess model fit, and detect multicollinearity. All code runs directly in Google Colab without any local setup.

Open In Colab

Chapter Overview

This chapter extends bivariate regression to the more realistic case where we want to predict an outcome using multiple explanatory variables simultaneously. Multiple regression allows us to estimate the partial effect of each variable while controlling for others—a crucial feature for empirical economic analysis.

What you’ll learn:

  • Extend bivariate regression concepts to multiple regression with several regressors
  • Interpret pairwise correlations and use them for exploratory data analysis
  • Understand the ordinary least squares (OLS) method for multiple regression
  • Interpret partial effects: how one regressor affects \(y\) while holding others constant
  • Distinguish between partial effects and total effects
  • Evaluate model fit using R-squared and adjusted R-squared
  • Understand information criteria (AIC, BIC) for model selection
  • Recognize when regression coefficients cannot be estimated (perfect collinearity)

Dataset used:

  • AED_HOUSE.DTA: 29 houses sold in Davis, California (1999) with price, size, bedrooms, bathrooms, lot size, age, and month sold

Key economic question: What is the effect of house size on price after controlling for other characteristics like bedrooms, bathrooms, and age?

Chapter outline:

  • 10.1 Example - House Price and Characteristics
  • 10.2 Two-Way Scatterplots
  • 10.3 Correlation Analysis
  • 10.4 Multiple Regression Estimation
  • 10.5 Partial Effects - The FWL Theorem
  • 10.6 Model Fit Statistics
  • 10.7 Model Comparison
  • 10.8 Inestimable Models and Multicollinearity
  • Key Takeaways
  • Practice Exercises
  • Case Studies

Key Concepts

Six core ideas anchor this chapter. Skim them before you start, and come back when a term feels fuzzy. Each entry pairs a concrete example using the chapter’s data with a non-technical analogy. Click a panel to expand it.

Multiple Regression Model: A regression with two or more explanatory variables, \(\hat{y} = b_1 + b_2 x_2 + b_3 x_3 + \cdots + b_k x_k\). Unlike bivariate regression, each slope \(b_j\) is a partial coefficient — it isolates the effect of \(x_j\) on \(y\) while statistically holding the other regressors fixed.

For the 29 Davis houses, the chapter fits \(\widehat{\text{price}} = b_1 + b_2 \cdot \text{size} + b_3 \cdot \text{bedrooms} + b_4 \cdot \text{bathrooms} + b_5 \cdot \text{lotsize} + b_6 \cdot \text{age} + b_7 \cdot \text{monthsold}\). The estimated size coefficient is $68.37/sq ft (down from $73.77 in the bivariate version) — a partial effect that excludes the influence of bedrooms, bathrooms, and the rest.

An orchestra plays a chord: the cello, viola, and violin all sound at once. Bivariate regression listens to one instrument as if it were the whole sound. Multiple regression is a sound engineer’s isolation track — turning each instrument up against a backdrop of the others, so you hear what that one instrument adds on its own.

Pairwise Correlation Matrix: A square table whose \((i, j)\) entry is the correlation \(r_{x_i x_j}\) between two columns of the dataset. It is the standard exploratory tool before fitting multiple regression — strong off-diagonal correlations signal candidate predictors and warn of likely multicollinearity.

The chapter’s correlation matrix on the Davis houses shows \(r(\text{price}, \text{size}) = 0.79\) (the strongest predictor) and \(r(\text{price}, \text{bedrooms}) = 0.43\). It also flags \(r(\text{size}, \text{bedrooms}) = 0.52\) — a moderate inter-regressor correlation that explains why the bedrooms coefficient drops from $23{,}667 in the bivariate fit to just $1{,}553 once size is in the model.

A wedding seating chart records, for every pair of guests, how well they get along. A pairwise correlation matrix is the same idea applied to data columns: for every pair of variables, one number summarising how often they “show up together”. Strong pairs sit close; weak pairs go to opposite ends of the room.

Control Variable: A regressor included in the model not because it is the main quantity of interest but to prevent its omission from biasing the focal coefficient. By adding the control, the regression isolates the partial effect of the focal variable while “holding the control fixed”.

In the Davis house regression, bedrooms looks like a strong predictor of price on its own ($23{,}667 per bedroom). Once size is added as a control variable, the bedrooms coefficient collapses to $1{,}553 — most of the apparent bedrooms effect was really houses with more bedrooms being bigger. size here is a textbook control: not the “treatment” of interest but essential for an honest estimate of bedrooms.

A laboratory experiment on plant growth holds temperature constant in every sample so the researcher can isolate the effect of fertiliser. Temperature is the lab’s control variable — not the thing being studied, but the thing being kept fixed so the rest of the comparison is fair. In regression, control variables play the same role with statistical instead of physical fixing.

Confounder: A variable that influences both the focal regressor and the outcome, creating an indirect channel through which the focal regressor appears to affect \(y\) even when it does not. Including the confounder as a control variable typically shrinks (or flips) the focal coefficient toward its true partial effect.

For the Davis houses, size is a confounder of the bedrooms–price relationship: bigger houses tend to have more bedrooms and fetch higher prices. That is why omitting size made bedrooms look like a $23{,}667-per-room driver, while including it brought the partial effect down to $1{,}553 — most of the original bedrooms coefficient was really size showing up in disguise.

Across cities, ice-cream sales and drowning deaths are tightly correlated — but ice cream isn’t drowning anyone. Summer is the confounder: it raises both sales and water-park traffic. Failing to control for the season makes ice cream look like a killer; controlling for it reveals the spurious link, just as size reveals bedrooms’ inflated coefficient.

Adjusted \(R^2\): A version of \(R^2\) that penalises for the number of regressors: \(\bar{R}^2 = 1 - \frac{\text{RSS}/(n-k)}{\text{TSS}/(n-1)}\). Adding a useless variable never decreases plain \(R^2\) but typically lowers \(\bar{R}^2\) — making it the better fit metric when comparing models of different sizes.

For the Davis houses, the simple regression on size alone gives \(R^2 = 0.617\) and \(\bar{R}^2 = 0.603\). The full model with all six regressors raises \(R^2\) to \(0.651\) — but \(\bar{R}^2\) actually falls to \(0.555\). Plain \(R^2\) rewards extra variables; adjusted \(R^2\) recognises that the extras barely earned their keep on only \(n = 29\) observations.

A football league lets teams sign extra players, but charges a salary-cap penalty for each one. A team with twenty players might score slightly more goals (raw \(R^2\) rises), yet still finish the season worse off after paying the penalty (adjusted \(R^2\) falls). Adjusted \(R^2\) is the league’s way of asking, “did those extra signings really help?”

Perfect Collinearity: The situation where one regressor is an exact linear combination of others (e.g., size_twice = 2 × size). When this happens, the OLS normal equations have no unique solution — software either drops a regressor automatically or returns an error.

The chapter constructs size_twice = 2 × size and tries to estimate price ~ size + size_twice + bedrooms. Because size_twice adds no new information beyond size, the regression is inestimable: the software drops one of the two collinear variables (pyfixest prints a warning naming it) and reports the rest. In contrast, the actual six predictors in the Davis dataset show only moderate VIFs, so all six can be estimated jointly.

Identical twins walk into a tailor’s shop wanting matching suits. The tailor has one set of measurements but two customers — there is no way to assign a “different” credit-card to each twin from the dataset alone, because the two are indistinguishable. Perfect collinearity is OLS facing identical twins among its regressors: distinct names, identical information.

Setup

First, we import the necessary Python packages and configure the environment for reproducibility. All data will stream directly from GitHub.

Setup: Import libraries and configure environment
# --- Libraries ---
import numpy as np                        # numerical operations
import pandas as pd                       # data manipulation
import matplotlib.pyplot as plt           # plotting
import seaborn as sns                     # statistical visualization
import pyfixest as pf                     # fast estimation with robust SEs
from scipy import stats                   # statistical distributions
import random
import os

# --- Reproducibility ---
RANDOM_SEED = 42
random.seed(RANDOM_SEED)
np.random.seed(RANDOM_SEED)
os.environ['PYTHONHASHSEED'] = str(RANDOM_SEED)

# --- Data source ---
GITHUB_DATA_URL = "https://raw.githubusercontent.com/quarcs-lab/data-open/master/AED/"

# --- Plotting style (dark theme matching book design) ---
plt.style.use('dark_background')
sns.set_style("darkgrid")
plt.rcParams.update({
    'axes.facecolor': '#1a2235',
    'figure.facecolor': '#12162c',
    'grid.color': '#3a4a6b',
    'figure.figsize': (10, 6),
    'text.color': 'white',
    'axes.labelcolor': 'white',
    'xtick.color': 'white',
    'ytick.color': 'white',
    'axes.edgecolor': '#1a2235',
})

print("Setup complete! Ready to analyze house price data.")
Setup complete! Ready to analyze house price data.

10.1 Example - House Price and Characteristics

We begin with a real estate dataset from Davis, California. Understanding house prices is a classic economic application because prices reflect both fundamental characteristics (size, bedrooms) and market conditions.

The dataset contains:

  • Price: Sale price in dollars
  • Size: House size in square feet
  • Bedrooms: Number of bedrooms
  • Bathrooms: Number of bathrooms
  • Lotsize: Size of lot (1=small, 2=medium, 3=large)
  • Age: House age in years
  • Monthsold: Month of year house was sold

Economic motivation: A simple regression of price on bedrooms might find a positive relationship, but is this because bedrooms directly add value, or because houses with more bedrooms tend to be larger? Multiple regression helps us disentangle these effects.

The code below loads the dataset from GitHub and displays the first ten observations together with summary statistics. Check each variable’s scale and range — price is in dollars, size in square feet — before we start modeling.

# Load house price data
data_house = pd.read_stata(GITHUB_DATA_URL + 'AED_HOUSE.DTA')

# Display first few observations
print("First 10 observations:")
print(data_house[['price', 'size', 'bedrooms', 'bathrooms', 'lotsize', 'age', 'monthsold']].head(10))

# Display data summary
print("\nSummary statistics:")
data_house.describe()
First 10 observations:
    price  size  bedrooms  bathrooms  lotsize   age  monthsold
0  204000  1400         3        2.0        1  31.0          7
1  212000  1600         3        3.0        2  33.0          5
2  213000  1800         3        2.0        2  51.0          4
3  220000  1600         3        2.0        1  49.0          4
4  224500  2100         4        2.5        2  47.0          6
5  229000  1700         4        2.5        2  35.0          3
6  230000  2100         4        2.0        2  34.0          8
7  233000  1700         3        2.0        1  40.0          6
8  235000  1700         4        2.0        2  29.0          7
9  235000  1600         3        2.0        3  35.0          5

Summary statistics:
price size bedrooms bathrooms lotsize age monthsold list
count 29.000000 29.000000 29.000000 29.000000 29.000000 29.000000 29.000000 29.000000
mean 253910.344828 1882.758621 3.793103 2.206897 2.137931 36.413792 5.965517 257824.137931
std 37390.710695 398.272130 0.675030 0.341144 0.693034 7.118975 1.679344 40860.264099
min 204000.000000 1400.000000 3.000000 2.000000 1.000000 23.000000 3.000000 199900.000000
25% 233000.000000 1600.000000 3.000000 2.000000 2.000000 31.000000 5.000000 239000.000000
50% 244000.000000 1800.000000 4.000000 2.000000 2.000000 35.000000 6.000000 245000.000000
75% 270000.000000 2000.000000 4.000000 2.500000 3.000000 39.000000 7.000000 269000.000000
max 375000.000000 3300.000000 6.000000 3.000000 3.000000 51.000000 8.000000 386000.000000

Interpreting the summary statistics: The 29 Davis houses sold for an average of about $253,910 (ranging from $204,000 to $375,000) and average about 1,883 square feet, with 3.8 bedrooms and 2.2 bathrooms; the typical house is about 36 years old. Keep this small sample size in mind — with only 29 observations, coefficients on six regressors will be imprecisely estimated.

Bivariate vs. Multiple Regression

Let’s compare a simple regression (price on bedrooms only) with a multiple regression (price on bedrooms AND size). This illustrates how controlling for other variables changes coefficient estimates.

Key insight: In the bivariate regression, the bedrooms coefficient captures both the direct effect of bedrooms and the indirect effect through correlation with size. In multiple regression, we isolate the partial effect of bedrooms holding size constant.

# Bivariate regression: price ~ bedrooms
model_bivariate = pf.feols('price ~ bedrooms', data=data_house)

# Key results: bivariate
print(f"Bivariate: bedrooms coef = ${model_bivariate.coef()['bedrooms']:,.2f}, R² = {model_bivariate._r2:.4f}")

# Full regression output
model_bivariate.summary()

# Multiple regression: price ~ bedrooms + size
model_multiple = pf.feols('price ~ bedrooms + size', data=data_house)

# Key results: multiple
print(f"Multiple: bedrooms coef = ${model_multiple.coef()['bedrooms']:,.2f}, R² = {model_multiple._r2:.4f}")

# Full regression output
model_multiple.summary()

# Compare bedrooms coefficient
# COEFFICIENT COMPARISON
print(f"Bedrooms coefficient (bivariate):  ${model_bivariate.coef()['bedrooms']:,.2f}")
print(f"Bedrooms coefficient (multiple):   ${model_multiple.coef()['bedrooms']:,.2f}")
print(f"Change: ${model_multiple.coef()['bedrooms'] - model_bivariate.coef()['bedrooms']:,.2f}")
Bivariate: bedrooms coef = $23,667.30, R² = 0.1826
###

Estimation:  OLS
Dep. var.: price
sample: None = all
Inference:  iid
Observations:  29

| Coefficient   |   Estimate |   Std. Error |   t value |   Pr(>|t|) |      2.5% |      97.5% |
|:--------------|-----------:|-------------:|----------:|-----------:|----------:|-----------:|
| Intercept     | 164137.838 |    37112.572 |     4.423 |      0.000 | 87989.131 | 240286.545 |
| bedrooms      |  23667.297 |     9637.976 |     2.456 |      0.021 |  3891.805 |  43442.790 |
---
RMSE: 33217.777 R2: 0.183 
Multiple: bedrooms coef = $1,553.46, R² = 0.6180
###

Estimation:  OLS
Dep. var.: price
sample: None = all
Inference:  iid
Observations:  29

| Coefficient   |   Estimate |   Std. Error |   t value |   Pr(>|t|) |       2.5% |      97.5% |
|:--------------|-----------:|-------------:|----------:|-----------:|-----------:|-----------:|
| Intercept     | 111690.856 |    27589.074 |     4.048 |      0.000 |  54980.702 | 168401.010 |
| bedrooms      |   1553.458 |     7846.866 |     0.198 |      0.845 | -14576.007 |  17682.923 |
| size          |     72.408 |       13.300 |     5.444 |      0.000 |     45.070 |     99.746 |
---
RMSE: 22706.95 R2: 0.618 
Bedrooms coefficient (bivariate):  $23,667.30
Bedrooms coefficient (multiple):   $1,553.46
Change: $-22,113.84

Key Concept 10.1: Partial Effects vs. Total Effects in Multiple Regression

In bivariate regression, the bedrooms coefficient ($23,667) captures both the direct effect of bedrooms and the indirect effect through correlation with size. In multiple regression, the bedrooms coefficient drops to $1,553 — the partial effect holding size constant — and is no longer statistically significant (\(t = 0.20\), \(p = 0.84\)). This dramatic change illustrates why controlling for confounders is essential for isolating individual variable effects.

10.2 Two-Way Scatterplots

Before running multiple regression, it’s useful to visualize pairwise relationships between variables. A scatterplot matrix shows all two-way scatterplots simultaneously.

What to look for:

  • Strong linear relationships (potential predictors of price)
  • Correlation between explanatory variables (potential multicollinearity)
  • Outliers or non-linear patterns

The diagonal shows the distribution of each variable using kernel density estimates (KDE).

# Create scatterplot matrix
plot_vars = ['price', 'size', 'bedrooms', 'age']
g = sns.pairplot(data_house[plot_vars], diag_kind='kde',
                 plot_kws={'alpha': 0.6, 's': 50})  # alpha = transparency, s = marker size
g.fig.suptitle('Figure 10.1: Scatterplot Matrix - House Price Data', 
               fontsize=14, fontweight='bold', y=1.00)
plt.tight_layout()
plt.show()

print("Scatterplot matrix created.")
print("Notice: Price shows strongest relationship with Size.")

Scatterplot matrix created.
Notice: Price shows strongest relationship with Size.

Key Concept 10.2: Exploratory Data Analysis with Scatterplot Matrices

Pairwise scatterplot matrices display all two-way relationships simultaneously, revealing linear associations, nonlinearities, clusters, and outliers before formal modeling. They also highlight potential multicollinearity: if two regressors are correlated (e.g., size and bedrooms, with a moderate \(r = 0.52\)), their individual effects may be hard to separate in a regression.

10.3 Correlation Analysis

The correlation coefficient measures the strength of linear association between two variables, ranging from -1 (perfect negative) to +1 (perfect positive).

Correlation matrix insights:

  • Price is most correlated with Size (r = 0.79), then Bedrooms (r = 0.43)
  • Size and Bedrooms are correlated (r = 0.52), which can cause multicollinearity
  • Correlation ≠ causation (merely shows association)

A correlation heatmap provides visual representation with color intensity showing strength of correlation.

# Calculate correlation matrix
corr_vars = ['price', 'size', 'bedrooms', 'bathrooms', 'lotsize', 'age', 'monthsold']
corr_matrix = data_house[corr_vars].corr()

print("Correlation Matrix:")
print(corr_matrix.round(3))

# Visualize correlation matrix as heatmap
fig, ax = plt.subplots(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, fmt='.3f', cmap='coolwarm', center=0,
            square=True, linewidths=1, cbar_kws={"shrink": 0.8})
ax.set_title('Correlation Matrix Heatmap', fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()

print(f"Price-Size correlation: {corr_matrix.loc['price', 'size']:.3f}")
print(f"Size-Bedrooms correlation: {corr_matrix.loc['size', 'bedrooms']:.3f}")
Correlation Matrix:
           price   size  bedrooms  bathrooms  lotsize    age  monthsold
price      1.000  0.786     0.427      0.330    0.153 -0.068     -0.210
size       0.786  1.000     0.518      0.316    0.112  0.077     -0.215
bedrooms   0.427  0.518     1.000      0.037    0.292 -0.026      0.183
bathrooms  0.330  0.316     0.037      1.000    0.102  0.037     -0.392
lotsize    0.153  0.112     0.292      0.102    1.000 -0.019     -0.057
age       -0.068  0.077    -0.026      0.037   -0.019  1.000     -0.366
monthsold -0.210 -0.215     0.183     -0.392   -0.057 -0.366      1.000

Price-Size correlation: 0.786
Size-Bedrooms correlation: 0.518

Key Concept 10.3: Correlation vs. Causation in Multivariate Analysis

High bivariate correlation (e.g., bedrooms-price, \(r = 0.43\)) may diminish or vanish after controlling for confounders. In our data, bedrooms correlate with price largely because bigger houses have more bedrooms. Multiple regression isolates each variable’s partial contribution, revealing that size — not bedrooms — accounts for most of the price variation.

10.4 Multiple Regression Estimation

Now we estimate the full multiple regression model with all available predictors. The regression equation is:

\[\widehat{\text{price}} = b_1 + b_2 \times \text{size} + b_3 \times \text{bedrooms} + b_4 \times \text{bathrooms} + b_5 \times \text{lotsize} + b_6 \times \text{age} + b_7 \times \text{monthsold}\]

Ordinary Least Squares (OLS) chooses coefficients \(b_1, ..., b_7\) to minimize the sum of squared residuals:

\[\min \sum_{i=1}^{n} (y_i - \widehat{y}_i)^2\]

where \(y_i\) is the actual price and \(\widehat{y}_i\) is the predicted price.

# Estimate full multiple regression model
model_full = pf.feols('price ~ size + bedrooms + bathrooms + lotsize + age + monthsold', data=data_house)

# Key results
r_squared_full = model_full._r2
adj_r_squared_full = model_full._adj_r2
size_coef = model_full.coef()['size']

print(f"Estimated equation: price = {model_full.coef()['Intercept']:,.0f} + {size_coef:.2f} x size + ...")
print(f"Size effect: each additional sq ft is associated with ${size_coef:,.2f} higher price")
print(f"R-squared: {r_squared_full:.4f} ({r_squared_full*100:.1f}% of variation explained)")
print(f"Adjusted R-squared: {adj_r_squared_full:.4f}")

# Full regression output
model_full.summary()
Estimated equation: price = 137,791 + 68.37 x size + ...
Size effect: each additional sq ft is associated with $68.37 higher price
R-squared: 0.6506 (65.1% of variation explained)
Adjusted R-squared: 0.5552
###

Estimation:  OLS
Dep. var.: price
sample: None = all
Inference:  iid
Observations:  29

| Coefficient   |   Estimate |   Std. Error |   t value |   Pr(>|t|) |       2.5% |      97.5% |
|:--------------|-----------:|-------------:|----------:|-----------:|-----------:|-----------:|
| Intercept     | 137791.066 |    61464.952 |     2.242 |      0.035 |  10320.557 | 265261.574 |
| size          |     68.369 |       15.389 |     4.443 |      0.000 |     36.454 |    100.285 |
| bedrooms      |   2685.315 |     9192.526 |     0.292 |      0.773 | -16378.816 |  21749.447 |
| bathrooms     |   6832.880 |    15721.192 |     0.435 |      0.668 | -25770.876 |  39436.636 |
| lotsize       |   2303.221 |     7226.535 |     0.319 |      0.753 | -12683.695 |  17290.138 |
| age           |   -833.039 |      719.335 |    -1.158 |      0.259 |  -2324.847 |    658.770 |
| monthsold     |  -2088.504 |     3520.898 |    -0.593 |      0.559 |  -9390.399 |   5213.392 |
---
RMSE: 21718.733 R2: 0.651 

Coefficient Interpretation with Confidence Intervals

Each regression coefficient represents a partial effect: the expected change in price when that variable increases by one unit, holding all other variables constant.

Example interpretation (Size coefficient):

  • Coefficient ≈ $68.37 per square foot
  • Interpretation: A one square foot increase in house size is associated with a $68.37 increase in price, holding bedrooms, bathrooms, lot size, age, and month sold constant.
  • Significance: This effect is statistically significant (\(t = 4.44\), \(p < 0.001\)) — size is the only regressor whose 95% confidence interval excludes zero.

The 95% confidence interval tells us the range of plausible values for each coefficient. Throughout this chapter we report classical (homoskedasticity-assuming) standard errors, as AED does before Chapter 12; Chapter 12 introduces heteroskedasticity-robust standard errors, which are preferred in practice.

# Display coefficients with 95% confidence intervals
conf_int = model_full.confint()
coef_table = pd.DataFrame({
    'Coefficient': model_full.coef(),
    'Std. Error': model_full.se(),
    'CI Lower': conf_int.iloc[:, 0],
    'CI Upper': conf_int.iloc[:, 1],
    't-statistic': model_full.tstat(),
    'p-value': model_full.pvalue()
})

print("Coefficients with 95% Confidence Intervals:")
coef_table

# Size is the only statistically significant predictor (p < 0.05)
Coefficients with 95% Confidence Intervals:
Coefficient Std. Error CI Lower CI Upper t-statistic p-value
Intercept 137791.065699 61464.951869 10320.557398 265261.573999 2.241783 0.035387
size 68.369419 15.389472 36.453608 100.285230 4.442610 0.000205
bedrooms 2685.315122 9192.525674 -16378.816300 21749.446543 0.292119 0.772932
bathrooms 6832.880015 15721.191544 -25770.875723 39436.635753 0.434629 0.668065
lotsize 2303.221371 7226.535205 -12683.695364 17290.138107 0.318717 0.752947
age -833.038602 719.334544 -2324.847139 658.769936 -1.158068 0.259254
monthsold -2088.503625 3520.897859 -9390.398871 5213.391620 -0.593174 0.559114

Interpreting the table: Size is the only statistically significant predictor (t = 4.44, p < 0.001), with a 95% confidence interval of roughly $36 to $100 per square foot. Every other coefficient has a confidence interval that comfortably includes zero — once size is controlled for, bedrooms, bathrooms, lot size, age, and month sold add little explanatory power in this small sample.

Key Concept 10.4: Interpreting Partial Effects in Multiple Regression

Each coefficient \(b_j\) measures the expected change in \(y\) when \(x_j\) increases by one unit, holding all other regressors constant. For example, a size coefficient of $68.37 means each additional square foot is associated with a $68.37 price increase, controlling for bedrooms, bathrooms, lot size, age, and month sold. Statistical significance is assessed through confidence intervals and \(t\)-tests.

10.5 Partial Effects - The FWL Theorem

The Frisch-Waugh-Lovell (FWL) Theorem states that the coefficient on any variable in multiple regression equals the coefficient from a bivariate regression of \(y\) on the residualized version of that variable.

Demonstration:

  1. Regress size on all other variables, obtain residuals \(\widetilde{\text{size}}\)
  2. Regress price on \(\widetilde{\text{size}}\) only
  3. The coefficient will exactly match the size coefficient from the full multiple regression

Intuition: The residual \(\widetilde{\text{size}}\) represents the variation in size that is not explained by other variables. This is why multiple regression isolates partial effects.

# Step 1: Regress size on all other variables
model_size_on_others = pf.feols('size ~ bedrooms + bathrooms + lotsize + age + monthsold', data=data_house)
resid_size = model_size_on_others._u_hat

# Step 2: Regress price on residualized size
data_house['resid_size'] = resid_size
model_price_on_resid = pf.feols('price ~ resid_size', data=data_house)

# Compare coefficients
# DEMONSTRATION: FWL THEOREM (Partial Effects)
print(f"Size coefficient from FULL multiple regression:  {model_full.coef()['size']:.10f}")
print(f"Coefficient on residualized size (bivariate):    {model_price_on_resid.coef()['resid_size']:.10f}")
print(f"Difference (numerical precision):                 {abs(model_full.coef()['size'] - model_price_on_resid.coef()['resid_size']):.15f}")
Size coefficient from FULL multiple regression:  68.3694189767
Coefficient on residualized size (bivariate):    68.3694189767
Difference (numerical precision):                 0.000000000000512

Key Concept 10.5: The Frisch-Waugh-Lovell (FWL) Theorem

The partial effect of \(x_j\) in a multiple regression equals the slope from a bivariate regression of \(y\) on \(\widetilde{x}_j\), where \(\widetilde{x}_j\) is the residual from regressing \(x_j\) on all other regressors. Intuitively, \(\widetilde{x}_j\) captures the variation in \(x_j\) that is independent of the other variables — this is exactly what multiple regression uses to estimate partial effects.

10.6 Model Fit Statistics

Several statistics summarize how well the regression model fits the data:

R-squared (\(R^2\)):

  • Fraction of variation in \(y\) explained by the regressors
  • Formula: \(R^2 = \frac{\text{Explained SS}}{\text{Total SS}} = 1 - \frac{\text{Residual SS}}{\text{Total SS}}\)
  • Range: 0 to 1 (higher is better fit)
  • Problem: Never decreases when adding variables (even irrelevant ones)

Adjusted R-squared (\(\bar{R}^2\)):

  • Penalizes model complexity (number of parameters \(k\))
  • Formula: \(\bar{R}^2 = 1 - \frac{\text{RSS}/(n-k)}{\text{TSS}/(n-1)}\)
  • Can decrease when adding unhelpful variables
  • Preferred for model comparison

Root MSE (Standard error of regression):

  • Typical size of prediction error
  • Formula: \(s_e = \sqrt{\frac{1}{n-k}\sum (y_i - \widehat{y}_i)^2}\)
  • Same units as \(y\) (dollars in our case)
  • Note: the RMSE printed in pyfixest summaries divides by \(n\) rather than \(n-k\), so it is slightly smaller than \(s_e\)
# Calculate and display model fit statistics
n = len(data_house)
k = len(model_full.coef())  # includes intercept
df = n - k

# MODEL FIT STATISTICS
print(f"Sample size (n):               {n}")
print(f"Number of parameters (k):      {k}")
print(f"Degrees of freedom (n-k):      {df}")
print(f"\nR-squared:                     {model_full._r2:.6f}")
print(f"Adjusted R-squared:            {model_full._adj_r2:.6f}")
print(f"Root MSE:                      ${np.sqrt(np.sum(model_full._u_hat**2) / (int(model_full._N) - len(model_full.coef()))):,.2f}")

# Verify R² = [Corr(y, ŷ)]²
predicted = model_full.predict()
corr_y_yhat = np.corrcoef(data_house['price'], predicted)[0, 1]
print(f"\nVerification: R² = [Corr(y, ŷ)]²")
print(f"  Correlation(y, ŷ):           {corr_y_yhat:.6f}")
print(f"  [Correlation(y, ŷ)]²:        {corr_y_yhat**2:.6f}")
print(f"  R² from model:                {model_full._r2:.6f}")
print(f"  Match: {np.isclose(corr_y_yhat**2, model_full._r2)}")
Sample size (n):               29
Number of parameters (k):      7
Degrees of freedom (n-k):      22

R-squared:                     0.650553
Adjusted R-squared:            0.555249
Root MSE:                      $24,935.73

Verification: R² = [Corr(y, ŷ)]²
  Correlation(y, ŷ):           0.806569
  [Correlation(y, ŷ)]²:        0.650553
  R² from model:                0.650553
  Match: True

Interpreting the fit statistics: The six regressors explain about 65% of the variation in price, but the adjusted \(R^2\) of 0.555 reveals that much of that gain is a mechanical reward for adding variables. The Root MSE of roughly $24,936 is the typical prediction error — sizable relative to an average price of about $253,910. The verification confirms that \(R^2\) equals the squared correlation between actual and fitted prices.

Information Criteria (AIC and BIC)

Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) are more sophisticated measures that penalize model complexity:

\[\text{AIC} = n \times \ln(\widehat{\sigma}_e^2) + n(1 + \ln 2\pi) + 2k\] \[\text{BIC} = n \times \ln(\widehat{\sigma}_e^2) + n(1 + \ln 2\pi) + k \times \ln(n)\]

where \(\widehat{\sigma}_e^2 = \text{RSS}/n\) (note the divisor is \(n\), not the \(n-k\) used for \(s_e\)).

Key points:

  • Lower values are better (unlike R²)
  • BIC penalizes complexity more heavily than AIC
  • Useful for comparing non-nested models
  • Different software packages may use different conventions (scaling by \(n\))
# Calculate information criteria
# INFORMATION CRITERIA

# Manual calculation of AIC and BIC (Stata convention)
rss = np.sum(model_full._u_hat ** 2)
aic_stata = n * np.log(rss/n) + n * (1 + np.log(2*np.pi)) + 2*k
bic_stata = n * np.log(rss/n) + n * (1 + np.log(2*np.pi)) + k*np.log(n)

print(f"AIC (Stata convention):        {aic_stata:.4f}")
print(f"BIC (Stata convention):        {bic_stata:.4f}")
AIC (Stata convention):        675.4824
BIC (Stata convention):        685.0535

Key Concept 10.6: Model Selection with Adjusted R-squared vs. Information Criteria

Adjusted \(R^2\) penalizes complexity mildly by dividing sums of squares by degrees of freedom. Information criteria (AIC, BIC) impose stronger penalties: BIC = \(n \ln(\hat{\sigma}_e^2) + k \ln(n)\) (omitting a constant that does not affect model comparisons). Smaller AIC/BIC values indicate better models. BIC is generally preferred because its penalty grows with sample size, favoring more parsimonious specifications.

10.7 Model Comparison

It’s often useful to compare multiple model specifications side-by-side. Here we compare:

  • Simple model: Price predicted by size only
  • Full model: Price predicted by all variables

Key comparison points:

  • How much does R² improve?
  • Does adjusted R² improve (accounting for added complexity)?
  • How do coefficient estimates change?

Economic interpretation: If adding 5 more variables only modestly improves fit, the simple model may be preferred (parsimony principle).

# Estimate simple model (size only)
model_simple = pf.feols('price ~ size', data=data_house)

# Information criteria (Stata convention). pyfixest's Feols has no ._aic/._bic,
# so compute them from each model's residual sum of squares.
def stata_ic(model):
    n_obs = int(model._N)
    n_par = len(model.coef())
    rss_m = np.sum(model._u_hat ** 2)
    const = n_obs * (1 + np.log(2 * np.pi))
    aic = n_obs * np.log(rss_m / n_obs) + const + 2 * n_par
    bic = n_obs * np.log(rss_m / n_obs) + const + n_par * np.log(n_obs)
    return aic, bic

aic_simple, bic_simple = stata_ic(model_simple)
aic_full, bic_full = stata_ic(model_full)

# Create comparison table
# MODEL COMPARISON: Simple vs. Full

comparison_stats = pd.DataFrame({
    'Model': ['Simple (size only)', 'Full (all variables)'],
    'R²': [model_simple._r2, model_full._r2],
    'Adj R²': [model_simple._adj_r2, model_full._adj_r2],
    'AIC': [aic_simple, aic_full],
    'BIC': [bic_simple, bic_full],
    'N': [n, n]
})

print(comparison_stats.to_string(index=False))

print(f"R² increases from {model_simple._r2:.3f} to {model_full._r2:.3f}")
print(f"Adj R² decreases from {model_simple._adj_r2:.3f} to {model_full._adj_r2:.3f}")
               Model       R²   Adj R²        AIC        BIC  N
  Simple (size only) 0.617453 0.603285 668.106844 670.841436 29
Full (all variables) 0.650553 0.555249 675.482401 685.053472 29
R² increases from 0.617 to 0.651
Adj R² decreases from 0.603 to 0.555

Key Concept 10.7: The Parsimony Principle

Simpler models are preferred unless additional variables meaningfully improve fit. In our house price example, adding five variables beyond size barely increased \(R^2\) (0.617 \(\to\) 0.651) while adjusted \(R^2\) actually fell (0.603 \(\to\) 0.555). Check adjusted \(R^2\), AIC, and BIC — if they don’t improve, the simpler model is preferred.

10.8 Inestimable Models and Multicollinearity

Perfect multicollinearity occurs when one regressor is an exact linear combination of others. In this case, OLS cannot estimate all coefficients (the model is “inestimable”).

Example: If we create size_twice = 2 × size and include both size and size_twice as regressors, the model is perfectly collinear.

Variance Inflation Factor (VIF) detects multicollinearity:

  • VIF measures how much a variable’s variance is inflated due to correlation with other regressors
  • Formula: \(VIF_j = \frac{1}{1 - R_j^2}\) where \(R_j^2\) is from regressing \(x_j\) on all other \(x\)’s
  • Rule of thumb: VIF > 10 indicates problematic multicollinearity

Consequences of high multicollinearity:

  • Large standard errors (imprecise estimates)
  • Unstable coefficients (small data changes → big estimate changes)
  • Difficulty interpreting individual effects
# Demonstrate perfect multicollinearity
# DEMONSTRATION: Perfect Multicollinearity

# Create a perfectly collinear variable
data_house['size_twice'] = 2 * data_house['size']

print("Creating variable: size_twice = 2 × size")
print("Attempting to estimate: price ~ size + size_twice + bedrooms\n")

try:
    model_collinear = pf.feols('price ~ size + size_twice + bedrooms', data=data_house)
    print("Model estimated (software automatically dropped one variable):")
    model_collinear.summary()
except Exception as e:
    print(f"Error: {type(e).__name__}")
    print(f"Message: {str(e)}")
Creating variable: size_twice = 2 × size
Attempting to estimate: price ~ size + size_twice + bedrooms

Model estimated (software automatically dropped one variable):
###

Estimation:  OLS
Dep. var.: price
sample: None = all
Inference:  iid
Observations:  29

| Coefficient   |   Estimate |   Std. Error |   t value |   Pr(>|t|) |       2.5% |      97.5% |
|:--------------|-----------:|-------------:|----------:|-----------:|-----------:|-----------:|
| Intercept     | 111690.856 |    27589.074 |     4.048 |      0.000 |  54980.702 | 168401.010 |
| size          |     72.408 |       13.300 |     5.444 |      0.000 |     45.070 |     99.746 |
| bedrooms      |   1553.458 |     7846.866 |     0.198 |      0.845 | -14576.007 |  17682.923 |
---
RMSE: 22706.95 R2: 0.618 

Interpreting the output: pyfixest warns that size_twice was dropped due to multicollinearity — it carries no information beyond size itself. Notice that the remaining estimates are identical to the earlier price ~ bedrooms + size regression: dropping the redundant regressor restores an estimable model without changing anything else.

Variance Inflation Factors (VIF)

Now let’s calculate VIF for each variable in our full model to check for multicollinearity problems.

# Calculate VIF for each variable in the full model
from statsmodels.stats.outliers_influence import variance_inflation_factor

X = data_house[['size', 'bedrooms', 'bathrooms', 'lotsize', 'age', 'monthsold']].copy()
X.insert(0, 'const', 1.0)  # VIF requires the intercept column in the design matrix
vif_data = pd.DataFrame()
vif_data["Variable"] = X.columns[1:]
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(1, X.shape[1])]

# VARIANCE INFLATION FACTORS (VIF)
print(vif_data.to_string(index=False))


max_vif = vif_data['VIF'].max()
if max_vif > 10:
    print(f"\nWarning: Maximum VIF = {max_vif:.2f} indicates multicollinearity issues.")
elif max_vif > 5:
    print(f"\nNote: Maximum VIF = {max_vif:.2f} shows moderate multicollinearity.")
else:
    print(f"\nGood: Maximum VIF = {max_vif:.2f} - no serious multicollinearity detected.")
 Variable      VIF
     size 1.691691
 bedrooms 1.733920
bathrooms 1.295270
  lotsize 1.129490
      age 1.180893
monthsold 1.574345

Good: Maximum VIF = 1.73 - no serious multicollinearity detected.

Key Concept 10.8: Detecting Multicollinearity with VIF

The Variance Inflation Factor (VIF) quantifies how much a coefficient’s variance is inflated due to correlation with other regressors. VIF\(_j = 1/(1 - R_j^2)\), where \(R_j^2\) is from regressing \(x_j\) on all other regressors. A VIF above 5 suggests moderate concern; above 10 indicates severe multicollinearity that inflates standard errors and destabilizes estimates. Perfect collinearity (VIF \(\to \infty\)) makes coefficients inestimable.

Visualization: Actual vs. Predicted Values

A plot of actual vs. predicted values helps visualize model fit. Points close to the 45-degree line indicate accurate predictions.

# Create actual vs predicted plot
fig, ax = plt.subplots(figsize=(10, 6))
ax.scatter(data_house['price'], model_full.predict(), alpha=0.6, s=50, color='#22d3ee')  # alpha = transparency, s = marker size
ax.plot([data_house['price'].min(), data_house['price'].max()],
        [data_house['price'].min(), data_house['price'].max()],
        'r--', linewidth=2, label='Perfect prediction (45° line)')
ax.set_xlabel('Actual Price ($)', fontsize=12)
ax.set_ylabel('Predicted Price ($)', fontsize=12)
ax.set_title('Actual vs Predicted House Prices', fontsize=14, fontweight='bold')
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print("Points close to the red line indicate accurate predictions.")

Points close to the red line indicate accurate predictions.

What to look for in this actual vs. predicted plot:

  • Alignment with 45-degree line: Points close to the red dashed line indicate accurate predictions
  • Systematic deviations: Clusters above or below the line suggest the model over- or under-predicts for certain price ranges
  • Spread: Wider scatter around the line means larger prediction errors

Visualization: Residual Plot

A residual plot (residuals vs. fitted values) helps diagnose model problems:

  • Random scatter around zero: Good (model assumptions satisfied)
  • Patterns (curves, funnels): Bad (model misspecification or heteroskedasticity)
  • Outliers: Investigate unusual observations
# Create residual plot
fig, ax = plt.subplots(figsize=(10, 6))
ax.scatter(model_full.predict(), model_full._u_hat, alpha=0.6, s=50, color='#22d3ee')  # alpha = transparency, s = marker size
ax.axhline(y=0, color='red', linestyle='--', linewidth=2, label='Zero residual line')
ax.set_xlabel('Fitted values ($)', fontsize=12)
ax.set_ylabel('Residuals ($)', fontsize=12)
ax.set_title('Residual Plot: Multiple Regression', fontsize=14, fontweight='bold')
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()

print("Random scatter around zero suggests model assumptions are reasonable.")

Random scatter around zero suggests model assumptions are reasonable.

What to look for in this residual plot:

  • Random scatter: Residuals should appear randomly distributed around the zero line (no pattern)
  • Funnel shape: A widening or narrowing pattern suggests heteroskedasticity (non-constant error variance)
  • Outliers: Points far from the zero line may indicate unusual observations worth investigating

Visualization: Coefficient Plot with Confidence Intervals

A coefficient plot displays estimated coefficients with their 95% confidence intervals. This makes it easy to see:

  • Which coefficients are significantly different from zero (CI doesn’t include zero)
  • Relative magnitude of effects
  • Precision of estimates (narrow vs. wide CIs)
# Create coefficient plot with confidence intervals
fig, ax = plt.subplots(figsize=(10, 8))

# Exclude intercept for better visualization
params_no_int = model_full.coef()[1:]
ci_no_int = conf_int.iloc[1:, :]

y_pos = np.arange(len(params_no_int))
ax.errorbar(params_no_int.values, y_pos,
            xerr=[params_no_int.values - ci_no_int.iloc[:, 0].values,
                  ci_no_int.iloc[:, 1].values - params_no_int.values],
            fmt='o', markersize=8, capsize=5, capthick=2, linewidth=2)
ax.set_yticks(y_pos)
ax.set_yticklabels(params_no_int.index)
ax.axvline(x=0, color='red', linestyle='--', linewidth=1, alpha=0.5, label='Zero effect')
ax.set_xlabel('Coefficient Value', fontsize=12)
ax.set_title('Coefficient Estimates with 95% Confidence Intervals',
             fontsize=14, fontweight='bold')
ax.legend()
ax.grid(True, alpha=0.3, axis='x')
plt.tight_layout()
plt.show()

print("Coefficients whose CI crosses zero are not statistically significant.")

Coefficients whose CI crosses zero are not statistically significant.

What to look for in this coefficient plot:

  • Zero crossing: Coefficients whose error bars cross the red dashed line are not statistically significant at 5%
  • Relative magnitude: Raw coefficients are not comparable across variables with different units — the size coefficient ($68 per square foot) sits nearest zero on this scale, while per-unit coefficients like bathrooms look larger; yet size is the only effect estimated precisely enough for its interval to exclude zero
  • Precision: Narrow error bars indicate more precisely estimated coefficients

Key Takeaways

Data Exploration and Correlation:

  • Multiple regression extends bivariate regression to include several regressors simultaneously
  • Summary statistics and two-way scatterplots provide initial overview of variable relationships
  • Correlation matrices reveal pairwise associations (e.g., price-size correlation of 0.79)
  • Correlation can be misleading — bedrooms correlate with price, but may simply reflect house size

OLS Estimation:

  • Multiple regression model: \(\widehat{y} = b_1 + b_2 x_2 + b_3 x_3 + \cdots + b_k x_k\)
  • OLS minimizes the sum of squared residuals: \(\min \sum (y_i - \widehat{y}_i)^2\)
  • The coefficient \(b_j\) can be computed by bivariate regression of \(y\) on \(\widetilde{x}_j\) (residualized \(x_j\))

Partial Effects:

  • Each coefficient \(b_j\) measures the partial effect: the change in \(\widehat{y}\) when \(x_j\) changes by one unit, holding all other regressors constant
  • Partial effects differ from total effects (which allow other regressors to vary)
  • OLS measures association, not causation — use careful language (“associated with”)

Model Fit:

  • \(R^2\) measures the fraction of variation in \(y\) explained by all regressors (0 to 1)
  • Adjusted \(R^2\) penalizes model complexity: can decrease when adding weak regressors
  • Standard error of regression \(s_e\) measures typical prediction error in units of \(y\)
  • Example: Adding 5 regressors increased \(R^2\) from 0.617 to 0.651 but decreased \(\bar{R}^2\) from 0.603 to 0.555

Information Criteria:

  • AIC and BIC penalize larger models more heavily than adjusted \(R^2\)
  • Smaller values indicate better models
  • BIC is generally preferred (stronger complexity penalty than AIC)

Multicollinearity:

  • Perfect collinearity makes some coefficients inestimable (the software drops one of the collinear variables, as pyfixest did with size_twice)
  • VIF detects multicollinearity: VIF > 10 indicates problematic, VIF > 5 moderate concern
  • High multicollinearity inflates standard errors and makes estimates unstable

Python Libraries and Code:

This single code block reproduces the core workflow of Chapter 10. It is self-contained — copy it into an empty notebook and run it to review the complete pipeline from data exploration and correlation analysis through multiple regression estimation, partial effects, model comparison, and multicollinearity diagnostics.

# =============================================================================
# CHAPTER 10 CHEAT SHEET: Data Summary for Multiple Regression
# =============================================================================

# --- Libraries ---
import numpy as np                                          # numerical operations
import pandas as pd                                         # data loading and manipulation
import matplotlib.pyplot as plt                             # creating plots and visualizations
import seaborn as sns                                       # statistical visualization (heatmaps, pairplots)
import pyfixest as pf                                       # fast estimation with robust SEs
from statsmodels.stats.outliers_influence import variance_inflation_factor  # multicollinearity detection

# =============================================================================
# STEP 1: Load data and explore
# =============================================================================
# pd.read_stata() reads Stata .dta files directly from a URL
url = "https://raw.githubusercontent.com/quarcs-lab/data-open/master/AED/AED_HOUSE.DTA"
data_house = pd.read_stata(url)

print(f"Dataset: {data_house.shape[0]} observations, {data_house.shape[1]} variables")
print(data_house[['price', 'size', 'bedrooms', 'bathrooms', 'lotsize', 'age']].describe().round(2))

# =============================================================================
# STEP 2: Partial effects vs. total effects — why controls matter
# =============================================================================
# Bivariate regression captures TOTAL effect (direct + indirect through size)
model_bivariate = pf.feols('price ~ bedrooms', data=data_house)

# Multiple regression isolates the PARTIAL effect (holding size constant)
model_partial = pf.feols('price ~ bedrooms + size', data=data_house)

print(f"Bedrooms coefficient (bivariate):  ${model_bivariate.coef()['bedrooms']:,.2f}")
print(f"Bedrooms coefficient (multiple):   ${model_partial.coef()['bedrooms']:,.2f}")
print(f"Change: ${model_partial.coef()['bedrooms'] - model_bivariate.coef()['bedrooms']:,.2f}")
# The coefficient drops from ~$23,667 to ~$1,553 once we control for size

# =============================================================================
# STEP 3: Correlation matrix — check pairwise associations before modeling
# =============================================================================
# High correlations between regressors signal potential multicollinearity
corr_vars = ['price', 'size', 'bedrooms', 'bathrooms', 'lotsize', 'age']
corr_matrix = data_house[corr_vars].corr()

fig, ax = plt.subplots(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, fmt='.3f', cmap='coolwarm', center=0,
            square=True, linewidths=1)
ax.set_title('Correlation Matrix Heatmap')
plt.tight_layout()
plt.show()

print(f"Price-Size correlation: {corr_matrix.loc['price', 'size']:.3f}")
print(f"Size-Bedrooms correlation: {corr_matrix.loc['size', 'bedrooms']:.3f}")

# =============================================================================
# STEP 4: Full multiple regression — estimate partial effects
# =============================================================================
# Each coefficient measures the change in price for a one-unit change in that
# variable, holding ALL other regressors constant
model_full = pf.feols('price ~ size + bedrooms + bathrooms + lotsize + age + monthsold', data=data_house)

size_coef = model_full.coef()['size']
print(f"Size effect: each additional sq ft is associated with ${size_coef:,.2f} higher price")
print(f"R-squared: {model_full._r2:.4f} ({model_full._r2*100:.1f}% of variation explained)")
print(f"Adjusted R-squared: {model_full._adj_r2:.4f}")

# Full regression table (coefficients, std errors, t-stats, p-values)
model_full.summary()

# =============================================================================
# STEP 5: FWL theorem — how "holding constant" actually works
# =============================================================================
# Step A: Regress size on all other regressors, keep residuals
model_size_on_others = pf.feols('size ~ bedrooms + bathrooms + lotsize + age + monthsold', data=data_house)
resid_size = model_size_on_others._u_hat

# Step B: Regress price on those residuals — the slope matches the full model
data_house['resid_size'] = resid_size
model_fwl = pf.feols('price ~ resid_size', data=data_house)

print(f"Size coef from FULL regression:     {model_full.coef()['size']:.10f}")
print(f"Coef from FWL residual regression:  {model_fwl.coef()['resid_size']:.10f}")
# These are identical — the FWL theorem in action

# =============================================================================
# STEP 6: Model comparison — parsimony vs. complexity
# =============================================================================
# Compare simple (size only) vs. full model using fit statistics
model_simple = pf.feols('price ~ size', data=data_house)

# pyfixest has no ._aic/._bic; compute them (Stata convention) from each model's RSS
def stata_ic(model):
    n_obs, n_par = int(model._N), len(model.coef())
    rss_m = np.sum(model._u_hat ** 2)
    const = n_obs * (1 + np.log(2 * np.pi))
    return (n_obs * np.log(rss_m / n_obs) + const + 2 * n_par,
            n_obs * np.log(rss_m / n_obs) + const + n_par * np.log(n_obs))

comparison = pd.DataFrame({
    'Model': ['Size only', 'Full (all variables)'],
    'R²':     [model_simple._r2,     model_full._r2],
    'Adj R²': [model_simple._adj_r2, model_full._adj_r2],
    'AIC':    [stata_ic(model_simple)[0], stata_ic(model_full)[0]],
    'BIC':    [stata_ic(model_simple)[1], stata_ic(model_full)[1]],
})
print(comparison.to_string(index=False))
# Adj R² DECREASES when adding 5 weak predictors — parsimony wins

# =============================================================================
# STEP 7: VIF — detect multicollinearity
# =============================================================================
# VIF_j = 1 / (1 - R²_j), where R²_j is from regressing x_j on all other x's
# VIF > 5: moderate concern; VIF > 10: severe multicollinearity
X = data_house[['size', 'bedrooms', 'bathrooms', 'lotsize', 'age', 'monthsold']].copy()
X.insert(0, 'const', 1.0)  # VIF requires the intercept column in the design matrix
vif_data = pd.DataFrame({
    'Variable': X.columns[1:],
    'VIF': [variance_inflation_factor(X.values, i) for i in range(1, X.shape[1])]
})
print(vif_data.to_string(index=False))

# =============================================================================
# STEP 8: Diagnostics — actual vs. predicted and residual plots
# =============================================================================
# Good fit: points hug the 45° line; residuals scatter randomly around zero
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

# Actual vs. predicted
axes[0].scatter(data_house['price'], model_full.predict(), alpha=0.7, s=50)
axes[0].plot([data_house['price'].min(), data_house['price'].max()],
             [data_house['price'].min(), data_house['price'].max()],
             'r--', linewidth=2, label='Perfect prediction (45° line)')
axes[0].set_xlabel('Actual Price ($)')
axes[0].set_ylabel('Predicted Price ($)')
axes[0].set_title('Actual vs Predicted')
axes[0].legend()
axes[0].grid(True, alpha=0.3)

# Residual plot
axes[1].scatter(model_full.predict(), model_full._u_hat, alpha=0.7, s=50)
axes[1].axhline(y=0, color='red', linestyle='--', linewidth=2)
axes[1].set_xlabel('Fitted Values ($)')
axes[1].set_ylabel('Residuals ($)')
axes[1].set_title('Residual Plot')
axes[1].grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Try it yourself! Copy this code into an empty Google Colab notebook and run it: Open Colab

Python tools used: pyfixest (feols), statsmodels (VIF), seaborn (pairplot, heatmap), pandas (DataFrames), matplotlib (coefficient plots, diagnostics)

Next Steps:

  • Chapter 11: Statistical inference for multiple regression — hypothesis tests, confidence intervals, and overall F-tests for model significance
  • Chapter 12: Further topics in multiple regression — robust standard errors and prediction intervals
  • Chapter 13: Case studies applying the multiple regression toolkit to real research questions
  • Chapters 14-15: Richer specifications — regression with indicator variables and with transformed variables

Congratulations on completing Chapter 10! You now have the tools to estimate and evaluate multiple regression models, a foundation for the rest of the course.

Common Mistakes to Avoid

  • Omitting relevant variables: Leads to omitted variable bias in the remaining coefficients
  • Including too many variables: Can cause multicollinearity and inflate standard errors
  • Ignoring high VIF values: VIF > 10 suggests problematic multicollinearity

Practice Exercises

Test your understanding of multiple regression concepts with these exercises.


Exercise 1: Residuals and Fitted Values

A regression leads to the fitted equation \(\widehat{y} = 2 + 3x_2 + 4x_3\).

  1. What is the predicted value for observation \((x_2, x_3) = (2, 1)\)?

  2. If the actual value is \(y = 9\), what is the residual?

  3. Is this observation over-predicted or under-predicted?


Exercise 2: Partial vs. Total Effects

Suppose OLS regression on the same dataset leads to:

  • Bivariate: \(\widehat{y} = 6 + 5x_2\)
  • Multiple: \(\widehat{y} = 2 + 3x_2 + 4x_3\)
  1. Why does the coefficient on \(x_2\) change from 5 to 3?

  2. Which coefficient (5 or 3) represents the partial effect of \(x_2\)? Explain.

  3. Under what condition would the bivariate and multiple regression coefficients on \(x_2\) be equal?


Exercise 3: R-squared and Adjusted R-squared

OLS regression of \(y\) on \(x\) for a sample of size \(n = 53\) leads to Residual SS = 20 and Total SS = 50.

  1. Compute \(R^2\).

  2. Compute the standard error of the regression \(s_e\).

  3. Compute the correlation between \(y\) and \(\widehat{y}\).

  4. If we add 3 more regressors and the Residual SS drops to 18, compute the new \(R^2\) and adjusted \(R^2\). Does the model improve?


Exercise 4: Information Criteria

Consider two models estimated on \(n = 29\) observations:

  • Model A (2 parameters): AIC = 668.1, BIC = 670.8
  • Model B (7 parameters): AIC = 675.5, BIC = 685.1
  1. Which model is preferred by AIC? By BIC?

  2. Why does BIC penalize Model B more heavily than AIC?

  3. What economic reasoning supports the simpler model in this case?


Exercise 5: Multicollinearity Detection

Suppose a hypothetical house-price regression — one whose predictors overlap far more than the Davis houses used in this chapter (whose correctly centered VIFs are all below 2) — yields the following VIF values:

Variable VIF
Size 40.1
Bedrooms 57.8
Bathrooms 34.7
Age 21.0
  1. Which variables show the most severe multicollinearity?

  2. What consequences does high VIF have for coefficient estimates and inference?

  3. Suggest a strategy to reduce multicollinearity in this model.


Exercise 6: Model Building

You have data on employee wages with variables: education (years), experience (years), age, tenure (years at current firm), and gender.

  1. Why might including both age and experience create multicollinearity?

  2. Propose two alternative model specifications and explain the trade-offs.

  3. How would you use adjusted \(R^2\) and BIC to select the preferred specification?

Case Studies

Case Study 1: Multiple Regression for Cross-Country Productivity

In this case study, you will apply multiple regression techniques to investigate how human capital and physical capital jointly explain cross-country differences in labor productivity. Using data from 108 countries over 1990-2014, you will explore correlations, estimate multiple regression models, interpret partial effects, and compare model specifications.

Dataset: Mendez Convergence Clubs Data

  • Source: Mendez (2020), 108 countries, 1990-2014
  • Key variables:
    • lp — Labor productivity (GDP per worker)
    • kl — Physical capital per worker
    • h — Human capital index
    • GDPpc — Real GDP per capita
    • region — Geographic region

Research question: How do human capital and physical capital jointly explain cross-country productivity differences?

# Load the Mendez convergence clubs dataset
url = "https://raw.githubusercontent.com/quarcs-lab/mendez2020-convergence-clubs-code-data/master/assets/dat.csv"
dat = pd.read_csv(url)
dat_2014 = dat[dat['year'] == 2014].dropna(subset=['lp', 'kl', 'h']).copy()
print(f"Cross-section sample: {len(dat_2014)} countries (year 2014)")
dat_2014.head()
Cross-section sample: 108 countries (year 2014)
id country year Y K pop L s alpha_it GDPpc ... log_h_raw log_tfp_raw log_GDPpc log_lp log_ky log_h log_tfp isocode hi1990 region
24 1 Albania 2014 31420.0 112793 2.889676 0.883679 9.991768 NaN 10873.1910 ... 1.276222 6.135863 9.319217 10.490668 1.288642 1.287595 6.138250 ALB no Europe
49 2 Algeria 2014 497453.0 1721824 38.934334 11.080190 7.591877 NaN 12776.7180 ... 1.073113 7.230991 9.541094 10.777947 1.110468 1.048142 7.330354 DZA no Africa
74 3 Argentina 2014 859909.0 2148251 42.980026 18.098450 9.611316 0.567057 20007.1780 ... 1.245337 7.486544 9.934555 10.758764 0.860546 1.252455 7.509842 ARG no Americas
99 4 Armenia 2014 27411.0 44003 3.006154 1.201936 10.904130 0.366997 9118.2949 ... 1.348571 6.184609 9.194162 10.142462 0.389373 1.345050 6.269046 ARM no Asia
124 5 Australia 2014 1029705.0 3944976 23.622353 11.966840 11.666830 0.431143 43590.2810 ... 1.407353 7.549226 10.743364 11.405924 1.282674 1.407372 7.602583 AUS yes Oceania

5 rows × 29 columns

Task 1: Explore Productivity Data (Guided)

Explore the cross-country productivity dataset.

# Step 1: Create log-transformed variables
dat_2014['ln_lp'] = np.log(dat_2014['lp'])
dat_2014['ln_kl'] = np.log(dat_2014['kl'])

# Step 2: Summary statistics for key variables
print(dat_2014[['lp', 'kl', 'h', 'ln_lp', 'ln_kl']].describe())

# Step 3: Create a scatterplot matrix
sns.pairplot(dat_2014[['ln_lp', 'ln_kl', 'h']], diag_kind='kde')
plt.suptitle('Scatterplot Matrix: Productivity Variables', y=1.02)
plt.show()

Questions:

  • Which variable shows the strongest visual association with \(\ln(\text{lp})\)?
  • Do you see any nonlinear patterns or outliers?

Task 2: Correlation Analysis (Guided)

Compute and visualize the correlation matrix.

# Step 1: Compute correlation matrix
corr = dat_2014[['ln_lp', 'ln_kl', 'h']].corr()
print(corr)

# Step 2: Visualize as heatmap
sns.heatmap(corr, annot=True, fmt='.3f', cmap='coolwarm', center=0, square=True)
plt.title('Correlation Matrix: Productivity Variables')
plt.show()

Questions:

  • Which predictor is most strongly correlated with \(\ln(\text{lp})\)?
  • Are \(\ln(\text{kl})\) and h correlated with each other? What implications does this have for multiple regression?

Key Concept 10.9: Functional Form and Cross-Country Comparisons

When comparing countries with vastly different income levels, logarithmic transformations are essential. Using \(\ln(\text{lp})\) and \(\ln(\text{kl})\) compresses the scale so that both Luxembourg and Malawi can be meaningfully compared. Coefficients on log-transformed variables have elasticity interpretations, while coefficients on level variables (like h) represent semi-elasticities.

Task 3: Multiple Regression Estimation (Semi-guided)

Estimate bivariate and multiple regression models for labor productivity.

# Model 1: ln(lp) ~ ln(kl) only
model1 = pf.feols('ln_lp ~ ln_kl', data=dat_2014)
model1.summary()

# Model 2: ln(lp) ~ h only
model2 = pf.feols('ln_lp ~ h', data=dat_2014)
model2.summary()

# Model 3: ln(lp) ~ ln_kl + h (multiple regression)
model3 = pf.feols('ln_lp ~ ln_kl + h', data=dat_2014)
model3.summary()

Questions:

  • How do the coefficients on \(\ln(\text{kl})\) change between Model 1 and Model 3?
  • Does adding human capital improve the model? Compare \(R^2\) and adjusted \(R^2\).

Task 4: Interpret Partial Effects (Semi-guided)

Interpret the coefficients from the multiple regression model.

# Display coefficients with confidence intervals
print("Model 3 Coefficients:")
print(model3.coef())
print("\n95% Confidence Intervals:")
print(model3.confint())

Questions:

  • Interpret the coefficient on \(\ln(\text{kl})\) in Model 3. What does it mean in economic terms?
  • Interpret the coefficient on h. How does a one-unit increase in human capital relate to productivity?
  • Are both coefficients statistically significant? How do you know?

Task 5: Model Selection (Independent)

Compare models using fit statistics and information criteria.

Your tasks:

  1. Create a comparison table with \(R^2\), adjusted \(R^2\), AIC, and BIC for all three models
  2. Which model is preferred by adjusted \(R^2\)? By AIC? By BIC?
  3. Does the parsimony principle favor one model over another?
  4. Calculate VIF for Model 3 — is multicollinearity a concern?

Hint: Use model._r2 and model._adj_r2 for fit. pyfixest’s Feols has no ._aic/._bic, so compute them (Stata convention) from the residuals: AIC = n*np.log(rss/n) + n*(1+np.log(2*np.pi)) + 2*k and BIC = ... + k*np.log(n), with rss = np.sum(model._u_hat**2), n = int(model._N), k = len(model.coef()). Use variance_inflation_factor() from statsmodels for VIF.

Task 6: Development Policy Brief (Independent)

Write a 200-300 word policy brief summarizing your findings.

Your brief should address:

  1. What are the key determinants of cross-country labor productivity?
  2. What is the relative importance of physical capital vs. human capital?
  3. How do partial effects differ from bivariate associations?
  4. What policy implications follow from your analysis?
  5. What limitations should policymakers keep in mind (association vs. causation)?

Key Concept 10.10: Multiple Regression in Development Economics

Cross-country productivity regressions are central to development economics. By controlling for both physical capital (\(\ln(\text{kl})\)) and human capital (h), we can assess each factor’s partial contribution to productivity. This addresses a key policy question: should developing countries invest in machines or education? Multiple regression helps disentangle these effects, though causal claims require careful identification strategies.

What You’ve Learned

In this case study, you applied the full multiple regression toolkit to cross-country productivity data:

  • Explored data with scatterplot matrices and correlation analysis
  • Estimated bivariate and multiple regression models, observing how coefficients change
  • Interpreted partial effects of physical and human capital on productivity
  • Compared model specifications using adjusted \(R^2\), AIC, and BIC
  • Connected statistical findings to development policy questions

These skills form the foundation for more advanced empirical analysis in the chapters ahead.

Case Study 2: Multiple Satellite Predictors of Development

Research Question: Do satellite image embeddings improve our ability to predict municipal development beyond nighttime lights alone?

In Chapter 1, we introduced the DS4Bolivia project and estimated a bivariate regression of development on nighttime lights. Now we extend this analysis using Chapter 10’s multiple regression tools, adding satellite image embeddings as additional predictors to test whether they improve our ability to predict municipal development.

Background: Nighttime lights (NTL) are a well-established proxy for economic activity, but they capture only one dimension of a municipality’s characteristics — nocturnal luminosity. Daytime satellite imagery contains far richer information: building density, road networks, agricultural patterns, vegetation cover. Deep learning models can extract this information as 64-dimensional embedding vectors, where each dimension captures abstract visual patterns learned automatically from the data.

The key question for multiple regression: Does adding these satellite embeddings as extra regressors significantly improve explanatory power compared to NTL alone? And if so, which embeddings matter most?

Variables for this case study:

  • imds — Municipal Sustainable Development Index (0-100, outcome variable)
  • ln_NTLpc2017 — Log nighttime lights per capita (established predictor from Chapter 1)
  • A00, A10, A20, A30, A40 — Selected satellite image embedding dimensions (new predictors)
  • mun, dep — Municipality and department identifiers

Your Task: Use correlation analysis, multiple regression, the FWL theorem, and model comparison tools from Chapter 10 to evaluate whether satellite embeddings add predictive value beyond nighttime lights.

Load the DS4Bolivia Data

Let’s load the DS4Bolivia dataset and select the variables needed for our multiple regression analysis. We focus on the development index, nighttime lights, and five selected satellite embedding dimensions.

# Load the DS4Bolivia dataset
url_bol = "https://raw.githubusercontent.com/quarcs-lab/ds4bolivia/master/ds4bolivia_v20250523.csv"
bol = pd.read_csv(url_bol)

# Load and merge satellite image embeddings
url_sat = "https://raw.githubusercontent.com/quarcs-lab/ds4bolivia/master/satelliteEmbeddings/satelliteEmbeddings2017.csv"
sat = pd.read_csv(url_sat)
bol = bol.merge(sat[['asdf_id', 'A00', 'A10', 'A20', 'A30', 'A40']], on='asdf_id', how='left')

# Select key variables for this case study
key_vars = ['mun', 'dep', 'imds', 'ln_NTLpc2017', 'A00', 'A10', 'A20', 'A30', 'A40']
bol_sat = bol[key_vars].copy()

# DS4BOLIVIA DATASET — SATELLITE PREDICTORS
print(f"Dataset shape: {bol_sat.shape[0]} municipalities, {bol_sat.shape[1]} variables")
print(f"Departments: {bol['dep'].nunique()} unique departments")
print(f"Complete cases: {bol_sat.dropna().shape[0]}")
# FIRST 10 MUNICIPALITIES
print(bol_sat.head(10).to_string(index=False))
Dataset shape: 339 municipalities, 9 variables
Departments: 9 unique departments
Complete cases: 333
            mun        dep      imds  ln_NTLpc2017       A00      A10      A20       A30       A40
      Esmeralda      Oruro 53.200001     13.647187 -0.060433 0.276589 0.132779 -0.021201  0.027739
      Quillacas      Oruro 50.500000     13.821883 -0.104140 0.210359 0.143159  0.013397  0.006337
 Pampa Aullagas      Oruro 47.299999     13.406137 -0.107063 0.187294 0.159752 -0.035909  0.050702
      Llallagua     Potosí 58.099998     14.827301 -0.130482 0.200049 0.126726  0.074560 -0.078046
       Caripuyo     Potosí 43.500000     12.510039 -0.123298 0.201647 0.120984  0.068288 -0.076545
        Aiquile Cochabamba 50.400002     14.606511  0.011109 0.038179 0.079780 -0.025559 -0.138114
Villa de Sacaca     Potosí 42.200001     12.586779 -0.145689 0.140690 0.134057  0.084465 -0.090000
         Acasio     Potosí 42.099998     12.358384 -0.098959 0.108831 0.094674  0.068063 -0.096150
        Huanuni      Oruro 56.799999     14.774255 -0.141211 0.218252 0.110193  0.050727 -0.039102
    Huachacalla      Oruro 61.299999     15.581995 -0.139633 0.223694 0.132744  0.037684 -0.062146

Task 1: Explore Variables (Guided)

Objective: Understand the satellite embedding variables and how they compare to nighttime lights.

Instructions:

  1. Generate summary statistics for all predictor variables using describe()
  2. Compare the scale and distribution of NTL vs. embedding variables
  3. Check for missing values across all selected variables
  4. Consider: What are satellite embeddings? How do they differ from NTL?

Key insight: Unlike NTL (which has a clear physical interpretation — light intensity), embedding dimensions are abstract features extracted by neural networks. Dimension A00 doesn’t mean “vegetation” or “roads” — it captures a learned combination of visual patterns.

# Your code here: Explore the satellite predictor variables
#
# Step 1: Summary statistics for all numeric variables
# DESCRIPTIVE STATISTICS: DEVELOPMENT AND SATELLITE PREDICTORS
print(bol_sat[['imds', 'ln_NTLpc2017', 'A00', 'A10', 'A20', 'A30', 'A40']].describe().round(3))

# Step 2: Check for missing values
# MISSING VALUES
print(bol_sat.isnull().sum())

# Step 3: Compare variable ranges
# VARIABLE RANGES
for var in ['imds', 'ln_NTLpc2017', 'A00', 'A10', 'A20', 'A30', 'A40']:
    col = bol_sat[var].dropna()
    print(f"  {var:16s}  range: [{col.min():.3f}, {col.max():.3f}]  std: {col.std():.3f}")
          imds  ln_NTLpc2017      A00      A10      A20      A30      A40
count  339.000       333.000  339.000  339.000  339.000  339.000  339.000
mean    51.050        13.880   -0.050    0.087    0.093    0.005   -0.088
std      6.772         1.181    0.082    0.093    0.046    0.063    0.059
min     35.700         9.064   -0.165   -0.096   -0.057   -0.169   -0.205
25%     47.000        13.134   -0.120    0.012    0.061   -0.029   -0.131
50%     50.500        13.914   -0.071    0.094    0.102    0.001   -0.102
75%     54.850        14.774    0.016    0.164    0.126    0.037   -0.041
max     80.200        17.064    0.164    0.277    0.193    0.206    0.058
mun             0
dep             0
imds            0
ln_NTLpc2017    6
A00             0
A10             0
A20             0
A30             0
A40             0
dtype: int64
  imds              range: [35.700, 80.200]  std: 6.772
  ln_NTLpc2017      range: [9.064, 17.064]  std: 1.181
  A00               range: [-0.165, 0.164]  std: 0.082
  A10               range: [-0.096, 0.277]  std: 0.093
  A20               range: [-0.057, 0.193]  std: 0.046
  A30               range: [-0.169, 0.206]  std: 0.063
  A40               range: [-0.205, 0.058]  std: 0.059

Task 2: Correlation Analysis (Guided)

Objective: Compute and visualize the correlation structure among all predictors and the outcome variable.

Instructions:

  1. Compute the correlation matrix for imds, ln_NTLpc2017, and all five embedding variables
  2. Display the correlation matrix as a heatmap
  3. Identify: Which embeddings correlate most strongly with imds?
  4. Identify: Do the embeddings correlate with each other (potential multicollinearity)?
  5. Do the embeddings correlate with NTL, or do they capture different information?

Apply what you learned in section 10.3: Use seaborn.heatmap() with annotated correlation values.

# Your code here: Correlation analysis of satellite predictors
#
# Step 1: Compute correlation matrix
corr_vars = ['imds', 'ln_NTLpc2017', 'A00', 'A10', 'A20', 'A30', 'A40']
corr_sat = bol_sat[corr_vars].dropna().corr()

# CORRELATION MATRIX: DEVELOPMENT AND SATELLITE PREDICTORS
print(corr_sat.round(3))

# Step 2: Visualize as heatmap
fig, ax = plt.subplots(figsize=(10, 8))
sns.heatmap(corr_sat, annot=True, fmt='.3f', cmap='coolwarm', center=0,
            square=True, linewidths=1, cbar_kws={"shrink": 0.8})
ax.set_title('Correlation Matrix: IMDS, NTL, and Satellite Embeddings',
             fontsize=14, fontweight='bold')
plt.tight_layout()
plt.show()

# Step 3: Highlight strongest correlations with IMDS
# Correlations with IMDS
imds_corr = corr_sat['imds'].drop('imds').sort_values(key=abs, ascending=False)
for var, r in imds_corr.items():
    print(f"  {var:16s}  r = {r:+.3f}")
               imds  ln_NTLpc2017    A00    A10    A20    A30    A40
imds          1.000         0.589  0.221 -0.195 -0.102 -0.360 -0.077
ln_NTLpc2017  0.589         1.000  0.063 -0.033  0.238 -0.354  0.098
A00           0.221         0.063  1.000 -0.865 -0.658 -0.453 -0.690
A10          -0.195        -0.033 -0.865  1.000  0.636  0.223  0.715
A20          -0.102         0.238 -0.658  0.636  1.000  0.092  0.743
A30          -0.360        -0.354 -0.453  0.223  0.092  1.000  0.086
A40          -0.077         0.098 -0.690  0.715  0.743  0.086  1.000

  ln_NTLpc2017      r = +0.589
  A30               r = -0.360
  A00               r = +0.221
  A10               r = -0.195
  A20               r = -0.102
  A40               r = -0.077

Key Concept 10.11: High-Dimensional Satellite Features

Satellite embeddings are 64 abstract features extracted by deep learning models (convolutional neural networks) from daytime satellite imagery. Unlike handcrafted variables (e.g., NDVI for vegetation), each embedding dimension captures complex visual patterns — road density, building structures, agricultural layouts — learned automatically from the data. These features are not directly interpretable (dimension A00 doesn’t have a specific meaning), but they collectively encode rich information about a municipality’s physical landscape.

Task 3: Multiple Regression (Semi-guided)

Objective: Estimate a multiple regression model with NTL and satellite embeddings as predictors.

Instructions:

  1. Estimate the full model: imds ~ ln_NTLpc2017 + A00 + A10 + A20 + A30 + A40
  2. Display the regression summary
  3. Compare \(R^2\) with the bivariate NTL-only model from Chapter 1
  4. Interpret: How much does adding embeddings improve explanatory power?
  5. Which embedding coefficients are statistically significant?

Apply what you learned in section 10.4: Use pf.feols() from pyfixest and interpret partial effects.

# Your code here: Multiple regression with satellite predictors
#
# Step 1: Prepare data (drop missing values)
reg_data = bol_sat[['imds', 'ln_NTLpc2017', 'A00', 'A10', 'A20', 'A30', 'A40']].dropna()
print(f"Regression sample: {len(reg_data)} municipalities (complete cases)")

# Step 2: Bivariate model (NTL only — baseline from Chapter 1)
model_ntl = pf.feols('imds ~ ln_NTLpc2017', data=reg_data)
# MODEL 1: BIVARIATE — imds ~ ln_NTLpc2017
model_ntl.summary()

# Step 3: Multiple regression (NTL + all 5 embeddings)
model_full_sat = pf.feols('imds ~ ln_NTLpc2017 + A00 + A10 + A20 + A30 + A40', data=reg_data)
# MODEL 2: MULTIPLE — imds ~ ln_NTLpc2017 + A00 + A10 + A20 + A30 + A40
model_full_sat.summary()

# Step 4: Compare R-squared
# R-SQUARED COMPARISON
print(f"NTL only:           R² = {model_ntl._r2:.4f}")
print(f"NTL + 5 embeddings: R² = {model_full_sat._r2:.4f}")
print(f"Improvement:        ΔR² = {model_full_sat._r2 - model_ntl._r2:.4f}")
Regression sample: 333 municipalities (complete cases)
###

Estimation:  OLS
Dep. var.: imds
sample: None = all
Inference:  iid
Observations:  333

| Coefficient   |   Estimate |   Std. Error |   t value |   Pr(>|t|) |   2.5% |   97.5% |
|:--------------|-----------:|-------------:|----------:|-----------:|-------:|--------:|
| Intercept     |      4.377 |        3.544 |     1.235 |      0.218 | -2.593 |  11.348 |
| ln_NTLpc2017  |      3.370 |        0.254 |    13.249 |      0.000 |  2.870 |   3.871 |
---
RMSE: 5.456 R2: 0.347 
###

Estimation:  OLS
Dep. var.: imds
sample: None = all
Inference:  iid
Observations:  333

| Coefficient   |   Estimate |   Std. Error |   t value |   Pr(>|t|) |    2.5% |   97.5% |
|:--------------|-----------:|-------------:|----------:|-----------:|--------:|--------:|
| Intercept     |      9.435 |        3.848 |     2.452 |      0.015 |   1.866 |  17.004 |
| ln_NTLpc2017  |      3.440 |        0.272 |    12.645 |      0.000 |   2.905 |   3.976 |
| A00           |    -11.619 |        8.786 |    -1.323 |      0.187 | -28.903 |   5.664 |
| A10           |    -10.566 |        6.764 |    -1.562 |      0.119 | -23.871 |   2.740 |
| A20           |    -47.279 |       10.114 |    -4.675 |      0.000 | -67.176 | -27.382 |
| A30           |    -17.583 |        6.179 |    -2.846 |      0.005 | -29.739 |  -5.428 |
| A40           |     14.235 |        8.251 |     1.725 |      0.085 |  -1.998 |  30.467 |
---
RMSE: 5.089 R2: 0.432 
NTL only:           R² = 0.3466
NTL + 5 embeddings: R² = 0.4316
Improvement:        ΔR² = 0.0851

Task 4: Partial Effects via FWL (Semi-guided)

Objective: Demonstrate the Frisch-Waugh-Lovell theorem by showing that the NTL coefficient in the multiple regression equals the coefficient from regressing residualized IMDS on residualized NTL.

Instructions:

  1. Regress imds on all embedding variables (A00, A10, A20, A30, A40), save residuals \(e_y\)
  2. Regress ln_NTLpc2017 on all embedding variables, save residuals \(e_x\)
  3. Regress \(e_y\) on \(e_x\) (bivariate regression of residuals)
  4. Verify that the coefficient matches the NTL coefficient from the full multiple regression

Apply what you learned in section 10.5: This demonstrates that the partial effect of NTL is computed from the variation in NTL that is independent of the satellite embeddings.

# Your code here: FWL theorem demonstration
#
# Step 1: Regress imds on embeddings only, save residuals e_y
model_y_on_emb = pf.feols('imds ~ A00 + A10 + A20 + A30 + A40', data=reg_data)
e_y = model_y_on_emb._u_hat

# Step 2: Regress ln_NTLpc2017 on embeddings only, save residuals e_x
model_x_on_emb = pf.feols('ln_NTLpc2017 ~ A00 + A10 + A20 + A30 + A40', data=reg_data)
e_x = model_x_on_emb._u_hat

# Step 3: Regress e_y on e_x
fwl_data = pd.DataFrame({'e_y': e_y, 'e_x': e_x})
model_fwl = pf.feols('e_y ~ e_x', data=fwl_data)

# Step 4: Compare coefficients
# FWL THEOREM DEMONSTRATION
print(f"NTL coefficient from FULL multiple regression:  {model_full_sat.coef()['ln_NTLpc2017']:.10f}")
print(f"Coefficient from FWL residual regression:       {model_fwl.coef()['e_x']:.10f}")
print(f"Difference (numerical precision):                {abs(model_full_sat.coef()['ln_NTLpc2017'] - model_fwl.coef()['e_x']):.15f}")
# The coefficients are identical — confirming the FWL theorem
NTL coefficient from FULL multiple regression:  3.4402848933
Coefficient from FWL residual regression:       3.4402848933
Difference (numerical precision):                0.000000000000190

Task 5: Model Comparison (Independent)

Objective: Compare multiple model specifications using fit statistics and information criteria.

Your tasks:

  1. Estimate three models:
    • Model 1: imds ~ ln_NTLpc2017 (NTL only)
    • Model 2: imds ~ ln_NTLpc2017 + A00 + A10 (NTL + 2 embeddings)
    • Model 3: imds ~ ln_NTLpc2017 + A00 + A10 + A20 + A30 + A40 (NTL + 5 embeddings)
  2. Create a comparison table reporting \(R^2\), adjusted \(R^2\), AIC, and BIC for each model
  3. Compute fit statistics from model._r2 and model._adj_r2; pyfixest has no ._aic/._bic, so compute AIC/BIC from the residuals as n*np.log(rss/n) + n*(1+np.log(2*np.pi)) + 2*k and ... + k*np.log(n) (with rss = np.sum(model._u_hat**2), n = int(model._N), k = len(model.coef()))
  4. Which model is “best” by each criterion?
  5. Does the parsimony principle favor fewer or more embedding variables?

Hint: Use pd.DataFrame() to create a clean comparison table.

# Your code here: Model comparison
#
# Step 1: Estimate three models
# model_1 = pf.feols('imds ~ ln_NTLpc2017', data=reg_data)
# model_2 = pf.feols('imds ~ ln_NTLpc2017 + A00 + A10', data=reg_data)
# model_3 = pf.feols('imds ~ ln_NTLpc2017 + A00 + A10 + A20 + A30 + A40', data=reg_data)
#
# Step 2: Create comparison table
# comparison = pd.DataFrame({
#     'Model': ['NTL only', 'NTL + 2 embeddings', 'NTL + 5 embeddings'],
#     'R²': [model_1._r2, model_2._r2, model_3._r2],
#     'Adj R²': [model_1._adj_r2, model_2._adj_r2, model_3._adj_r2],
#     # pyfixest has no ._aic/._bic; compute them (Stata convention) from each model's RSS:
#     #   def stata_ic(m):
#     #       n_m, k_m = int(m._N), len(m.coef()); rss_m = np.sum(m._u_hat**2); c = n_m*(1+np.log(2*np.pi))
#     #       return (n_m*np.log(rss_m/n_m)+c+2*k_m, n_m*np.log(rss_m/n_m)+c+k_m*np.log(n_m))
#     'AIC': [stata_ic(model_1)[0], stata_ic(model_2)[0], stata_ic(model_3)[0]],
#     'BIC': [stata_ic(model_1)[1], stata_ic(model_2)[1], stata_ic(model_3)[1]],
#     'N': [len(reg_data)] * 3
# })
# print(comparison.to_string(index=False))
#
# Step 3: Interpret — which model is preferred by each criterion?

Key Concept 10.12: Incremental Predictive Power

When adding predictors to a regression model, \(R^2\) can only increase or stay the same — it never decreases. This makes \(R^2\) misleading for model comparison when models have different numbers of predictors. Adjusted \(R^2\) penalizes for additional variables, while AIC and BIC balance fit against complexity. In the satellite prediction context, adding all 64 embeddings would maximize \(R^2\) but might overfit; information criteria help identify the most parsimonious model.

Task 6: Policy Brief on Satellite Prediction (Independent)

Objective: Write a 200-300 word policy brief summarizing the value of satellite embeddings for development prediction.

Your brief should address:

  1. Improvement: How much does adding satellite embeddings improve development prediction compared to NTL alone?
  2. Complexity trade-off: Is the improvement worth the added model complexity? What do adjusted \(R^2\), AIC, and BIC suggest?
  3. Partial effects: After controlling for embeddings, does NTL remain a significant predictor? What does the FWL theorem reveal about NTL’s independent contribution?
  4. SDG monitoring implications: How could multi-source satellite data enhance SDG monitoring in data-scarce countries like Bolivia?
  5. Limitations: What can satellite data not capture about development? What are the risks of relying on abstract embedding features for policy decisions?

Connection to Research: The DS4Bolivia project uses all 64 embedding dimensions plus machine learning methods (Random Forest, XGBoost) to predict SDG indicators, achieving meaningful predictive accuracy. Your multiple regression analysis provides a transparent, interpretable baseline for comparison.

# Your code here: Additional analysis for the policy brief
#
# You might want to:
# 1. Create a summary table of key results across models
# 2. Generate a visualization comparing model fit
# 3. Calculate the percentage improvement in R² from adding embeddings
# 4. Discuss which embeddings contribute most to prediction
#
# Example:
# print("KEY RESULTS FOR POLICY BRIEF")
# print(f"NTL-only R²:          {model_1._r2:.4f}")
# print(f"NTL + 5 embed R²:     {model_3._r2:.4f}")
# print(f"R² improvement:       {(model_3._r2 - model_1._r2) / model_1._r2 * 100:.1f}%")
# print(f"NTL coef (bivariate): {model_1.coef()['ln_NTLpc2017']:.4f}")
# print(f"NTL coef (multiple):  {model_3.coef()['ln_NTLpc2017']:.4f}")

What You’ve Learned from This Case Study

Through this analysis of multiple satellite predictors of Bolivian municipal development, you’ve applied the full Chapter 10 toolkit to a cutting-edge research application:

  • Correlation analysis: Explored the correlation structure among NTL, satellite embeddings, and development outcomes
  • Multiple regression: Estimated models with multiple satellite predictors and interpreted partial effects
  • FWL theorem: Demonstrated that the partial effect of NTL equals the coefficient from regressing residualized IMDS on residualized NTL
  • Model comparison: Evaluated competing specifications using \(R^2\), adjusted \(R^2\), AIC, and BIC
  • Critical assessment: Weighed the predictive gains from additional satellite features against model complexity

Connection to upcoming chapters: In Chapter 11, we’ll test whether the satellite embeddings are statistically significant using F-tests for joint significance. In Chapter 12, we’ll address robust inference and prediction intervals.

This dataset returns throughout the textbook: Each subsequent chapter applies its specific econometric tools to the DS4Bolivia data, building progressively toward a comprehensive satellite-based development prediction framework.


Well done! You’ve now analyzed how multiple satellite data sources can predict development outcomes, moving from simple bivariate regression (Chapter 1) to the richer multiple regression framework of Chapter 10.