Chapter 17 — Panel Data, Time Series Data, Causation

Panel data has two souls: big-market teams earn more between themselves, and the same team earns more in some years than others within itself. Which variation does your estimator use?

Panel data variation decomposes into between (differences across individuals in their averages) and within (deviations from individual averages over time). This decomposition determines what each estimator identifies: pooled OLS uses both sources, fixed effects uses only within, and random effects uses a weighted combination. In the NBA example, between variation in revenue is large (big-market vs small-market), while within variation is smaller (year-to-year fluctuations).

Try this

Read log-revenue's three bars. Between SD ≈ 0.21; within SD ≈ 0.11. Most NBA revenue variation is across teams, not across seasons — big-market vs small-market dominates year-to-year swings.
Read wins' bars. Between and within are closer in magnitude. Team performance fluctuates substantially season-to-season — a lot of "wins variation" would survive de-meaning.
Connect to estimator choice. FE only uses the within bars. When within variation is small, FE estimates become imprecise — that's the price of removing all between-team confounding.

Take-away: What a panel estimator can identify depends on which kind of variation the data carries — inspect the decomposition before choosing pooled OLS, FE, or RE. Read §17.2 in the chapter →

Treat 286 team-seasons as independent and your SEs will be comically small. Cluster by team and reality returns — sometimes doubling the standard error.

Observations within the same individual (team, firm, country) are correlated over time — violating OLS's independence assumption. Default SEs dramatically understate uncertainty by treating all observations as independent. Cluster-robust SEs account for within-individual correlation, often producing SEs that are 2× or more larger than default. Always cluster by individual in panel data; with few clusters (G < 30), consider wild-bootstrap refinements.

Try this

Compare the three bars. Cluster SE ≈ 1.81× default; robust (HC1) SE ≈ 1.15× default. Robust corrects for heteroskedasticity but not serial correlation within a team — only clustering fixes both.
Check significance under each SE type. Wins stays significant (p = 0.0004) with cluster SEs. The coefficient is strong enough to survive the correction, but the CI is almost twice as wide — every panel inference should be reported with cluster SEs.
Imagine a borderline coefficient. With default SEs it would be "significant at 5%"; with cluster SEs it would not. This is where clustering changes published conclusions.

Take-away: Panel observations within the same individual are correlated — always cluster your standard errors by individual, or your t-statistics are fiction. Read §17.2 in the chapter →

Big-market teams win more and earn more — so does winning cause revenue, or is the Lakers' market the whole story? FE removes every persistent team trait and lets you see.

Fixed effects estimation controls for time-invariant individual characteristics by including individual-specific intercepts α_i. The within transformation (de-meaning y_it and x_it by the individual's average) eliminates α_i entirely and uses only variation within each individual over time. FE provides more credible causal estimates but cannot identify effects of time-invariant variables. FE is consistent whether or not α_i is correlated with regressors; random effects is more efficient but inconsistent if the uncorrelated-effects assumption fails (Key Concept 17.4). The Hausman test decides between them.

View

Try this

Start on Pooled OLS. Wins coef = 0.0068 — one extra win is associated with 0.68% higher revenue. Mixes within- and between-team variation — big-market teams win more, so the pooled estimate is contaminated.
Switch to Fixed Effects. Wins coef drops to 0.0045. FE strips out persistent team characteristics (market size, arena, brand) — the within effect of winning on revenue is smaller than pooled OLS suggested.
Toggle Both. The FE line is visibly flatter than the pooled line. The slope difference is the between-team confound in dollar terms.
Note FE within-R² ≈ 0.19. Only 19% of within-team revenue variation is explained by wins — much of the rest is idiosyncratic events (injuries, rule changes, playoff runs).

Take-away: FE removes bias from persistent individual characteristics at the cost of between-individual variation — credible at the price of precision. Read §17.3 in the chapter →

Two time series trending together will always have a high R² — even if they have nothing to do with each other. First differencing is the simplest defense.

A time series is stationary if its statistical properties (mean, variance, autocorrelation) are constant over time. Many economic series are non-stationary (trending) — and regressing non-stationary series on each other can produce spurious regressions: high R² and significant coefficients even when variables are unrelated. First differencing (Δy_t = y_t − y_t−1) typically removes trends and restores stationarity. Always check stationarity before interpreting a time-series regression.

View

Try this

Start on Levels. Both rates drift from ~14% in 1982 to ~2% by 2015. A shared downward trend — any regression of one on the other will inherit it as "fit," regardless of causality.
Switch to Changes. Both series fluctuate around zero with no drift. Stationary by construction — first differencing strips out the trend and delivers a series OLS can safely regress on.
Compare amplitudes. Monthly changes are ±1 percentage point; the level range is 12 points. The two views live on very different scales, which is why R² on levels looks so much larger than on changes.

Take-away: Trending series share a trend before they share anything else — difference them before regressing, or risk a spurious story. Read §17.5 in the chapter →

The ACF of the residuals is how you prove you have a stationarity problem — or prove that you fixed it.

The correlogram (ACF plot) reveals autocorrelation patterns in residuals. Slowly decaying autocorrelations (e.g., ρ₁ = 0.95, ρ₁₀ = 0.42) indicate non-stationarity and persistent shocks. With autocorrelation, default SEs are too small — HAC (Newey-West) SEs can be 3–8 times larger. Always check residual autocorrelation after estimating a time-series regression and use HAC SEs or model the dynamics explicitly.

Model residuals

Try this

Start on Levels residuals. Lag-1 ACF = 0.98; lag-10 ACF still > 0.80. Highly persistent residuals — the levels regression is spurious and the default SEs are invalid.
Switch to Changes. Lag-1 ACF drops to ~0.25 and most later lags sit inside the band. First differencing removed most of the serial correlation — a much better-specified model.
Switch to ADL(2,2). Lag-1 ACF ≈ 0.02 — essentially zero. The autoregressive and distributed-lag terms absorbed all the persistence — a textbook correctly-specified dynamic model.

Take-away: The residual ACF is the diagnostic that says "your model is misspecified" — and cleaning it up with differencing or lagged regressors is the cure. Read §17.6 in the chapter →

R² = 0.91 looks like a home run. But if the residuals are autocorrelated and the series are non-stationary, that 0.91 is lying to you.

First differencing transforms non-stationary trending series into stationary ones, eliminating spurious-regression problems. After differencing, the residual autocorrelation drops dramatically (from ρ₁ ≈ 0.95 to ρ₁ ≈ 0.25 in the interest-rate example). The coefficient interpretation changes from levels to changes: a 1-percentage-point change in the 1-year rate is associated with a 0.72-percentage-point change in the 10-year rate (Key Concept 17.7). R² on changes is always lower — but it's honest.

Regression

Try this

Stay on Levels. R² = 0.91, slope = 0.84. Looks impressive — but residual ACF₁ = 0.98 flags the R² as spurious, inflated by the shared downward trend.
Switch to Changes. R² = 0.57, slope = 0.72. Lower R² but honest — residual ACF₁ falls to 0.25, and the slope now measures genuine co-movement of monthly changes.
Compare SEs on levels. Default = 0.013; HAC = 0.045 (3.4× larger). Default SEs are untrustworthy with non-stationary data — any significance test built on them is fiction.
Interpret the changes slope (0.72). A 1 pp move in the 1-year rate is associated with a 0.72 pp move in the 10-year rate — the honest co-movement estimate.

Take-away: High R² on trending series is a warning, not a finding — first-difference before you interpret any time-series regression. Read §17.5 in the chapter →

A rate change today isn't a one-shot event — it propagates for months. Cumulative multipliers trace the full dynamic response from a single impulse.

An autoregressive distributed-lag model adds own lags and predictor lags to a regression, turning a static equation into a dynamic one. The impact multiplier γ₀ is the contemporaneous effect; cumulative multipliers sum γ₀ + γ₁ + γ₂ + ⋯ and show how the total effect builds (or reverses) over time. A well-specified ADL should leave no residual autocorrelation — it's the "model the dynamics explicitly" alternative to HAC SEs.

Try this

Read the impact multiplier (γ₀ = 0.86). A 1 pp shock to the 1-year rate immediately moves the 10-year rate by 0.86 pp — the contemporaneous response is large but not 1:1.
Read the 1-month cumulative (γ₀ + γ₁ = 0.31). The cumulative effect falls because γ₁ is negative — a partial reversal. Interest-rate transmission is not monotonic; the first month corrects toward the trend.
Read the 2-month cumulative (0.54). Recovers as γ₂ is positive. The dynamic path oscillates before settling — a good reminder that "the effect" of one variable on another is not a single number.
Check residual ACF₁ ≈ 0.02. The ADL(2,2) has absorbed virtually all the serial correlation — a well-specified dynamic model.

Take-away: ADL models make the dynamic response explicit — every static coefficient hides a multi-period adjustment path, and cumulative multipliers are how you recover it. Read §17.6 in the chapter →

Python Libraries and Code

You've explored the key concepts interactively — now reproduce them in Python. This self-contained code block covers everything you practiced above. Copy it into an empty notebook and run it.

# =============================================================================
# CHAPTER 17 CHEAT SHEET: Panel Data, Time Series Data, Causation
# =============================================================================

# --- Libraries ---
import pandas as pd                       # data loading and manipulation
import numpy as np                         # numerical operations
import matplotlib.pyplot as plt            # creating plots and visualizations
import statsmodels.api as sm               # statistical models and tools
from statsmodels.formula.api import ols    # OLS regression with R-style formulas
from statsmodels.tsa.stattools import acf  # autocorrelation function
from linearmodels.panel import PanelOLS    # fixed effects panel estimation

# =============================================================================
# STEP 1: Load panel data (NBA teams across seasons)
# =============================================================================
# Panel data: multiple individuals (teams) observed over multiple time periods
url_nba = "https://raw.githubusercontent.com/quarcs-lab/data-open/master/AED/AED_NBA.DTA"
data_nba = pd.read_stata(url_nba)

print(f"Panel: {data_nba['teamid'].nunique()} teams × {data_nba['season'].nunique()} seasons = {len(data_nba)} obs")

# =============================================================================
# STEP 2: Variance decomposition — between vs within variation
# =============================================================================
# Understanding which variation your estimator uses is the first step in panel analysis
overall_sd = data_nba['lnrevenue'].std()
between_sd = data_nba.groupby('teamid')['lnrevenue'].mean().std()
within_sd  = data_nba.groupby('teamid')['lnrevenue'].apply(lambda x: x - x.mean()).std()

print(f"\nVariance Decomposition of Log Revenue:")
print(f"  Overall SD:  {overall_sd:.4f}")
print(f"  Between SD:  {between_sd:.4f} (across teams)")
print(f"  Within SD:   {within_sd:.4f} (over time)")
print(f"  Between > Within → team characteristics dominate year-to-year swings")

# =============================================================================
# STEP 3: Pooled OLS with cluster-robust SEs
# =============================================================================
# Observations within the same team are correlated over time — default SEs
# dramatically understate uncertainty. Always cluster by individual in panel data.
model_pool = ols('lnrevenue ~ wins', data=data_nba).fit()
model_cluster = ols('lnrevenue ~ wins', data=data_nba).fit(
    cov_type='cluster', cov_kwds={'groups': data_nba['teamid']}
)

print(f"\nPooled OLS — wins coefficient: {model_pool.params['wins']:.6f}")
print(f"  Default SE:  {model_pool.bse['wins']:.6f}")
print(f"  Cluster SE:  {model_cluster.bse['wins']:.6f}")
print(f"  Ratio:       {model_cluster.bse['wins'] / model_pool.bse['wins']:.2f}x larger")

# =============================================================================
# STEP 4: Fixed effects — control for unobserved team characteristics
# =============================================================================
# FE uses only within-team variation (de-meaning), eliminating bias from
# persistent traits like market size, brand value, and arena quality.
data_panel = data_nba.set_index(['teamid', 'season'])
y = data_panel[['lnrevenue']]
X = data_panel[['wins']]

model_fe = PanelOLS(y, X, entity_effects=True).fit(cov_type='clustered', cluster_entity=True)

print(f"\nFixed Effects — wins coefficient: {model_fe.params['wins']:.6f}")
print(f"  Cluster SE:  {model_fe.std_errors['wins']:.6f}")
print(f"  R² (within): {model_fe.rsquared_within:.4f}")

print(f"\nComparison:")
print(f"  Pooled OLS coef: {model_pool.params['wins']:.6f}")
print(f"  Fixed Effects:   {model_fe.params['wins']:.6f}")
print(f"  FE is smaller → pooled OLS had positive omitted variable bias")

# =============================================================================
# STEP 5: Time series — levels vs first differences
# =============================================================================
# Non-stationary (trending) series produce spurious regressions with misleading R².
# First differencing removes trends and restores valid inference.
url_rates = "https://raw.githubusercontent.com/quarcs-lab/data-open/master/AED/AED_INTERESTRATES.DTA"
data_rates = pd.read_stata(url_rates)

# Regression in levels (potentially spurious)
model_levels = ols('gs10 ~ gs1', data=data_rates).fit()

# Regression in first differences (removes trends)
model_changes = ols('dgs10 ~ dgs1', data=data_rates).fit()

print(f"\nLevels regression:  gs1 coef = {model_levels.params['gs1']:.4f}, R² = {model_levels.rsquared:.4f}")
print(f"Changes regression: dgs1 coef = {model_changes.params['dgs1']:.4f}, R² = {model_changes.rsquared:.4f}")
print(f"R² drops after differencing — lower but honest (no spurious trend inflation)")

# =============================================================================
# STEP 6: Autocorrelation diagnostics — the smoking gun
# =============================================================================
# Slowly decaying ACF in residuals signals non-stationarity and invalid SEs.
# After differencing, autocorrelation should drop dramatically.
acf_levels  = acf(model_levels.resid.dropna(), nlags=5)
acf_changes = acf(model_changes.resid.dropna(), nlags=5)

print(f"\nResidual autocorrelation (lag 1):")
print(f"  Levels regression:  {acf_levels[1]:.4f} (high → non-stationary residuals)")
print(f"  Changes regression: {acf_changes[1]:.4f} (much lower → differencing worked)")

# HAC (Newey-West) SEs correct for autocorrelation without differencing
model_hac = ols('gs10 ~ gs1', data=data_rates).fit(cov_type='HAC', cov_kwds={'maxlags': 24})
print(f"\nDefault SE on gs1:   {model_levels.bse['gs1']:.4f}")
print(f"HAC SE on gs1:       {model_hac.bse['gs1']:.4f}")
print(f"HAC is {model_hac.bse['gs1'] / model_levels.bse['gs1']:.1f}x larger — default SEs are too small")

# =============================================================================
# STEP 7: ADL model — dynamic multipliers
# =============================================================================
# Autoregressive distributed lag models capture how effects build over time.
# Lagged dependent and independent variables model persistence and transmission.
data_rates['dgs10_lag1'] = data_rates['dgs10'].shift(1)
data_rates['dgs10_lag2'] = data_rates['dgs10'].shift(2)
data_rates['dgs1_lag1']  = data_rates['dgs1'].shift(1)
data_rates['dgs1_lag2']  = data_rates['dgs1'].shift(2)

model_adl = ols('dgs10 ~ dgs10_lag1 + dgs10_lag2 + dgs1 + dgs1_lag1 + dgs1_lag2',
                data=data_rates).fit()

print(f"\nADL(2,2) Model:")
print(f"  Impact multiplier (dgs1):       {model_adl.params['dgs1']:.4f}")
print(f"  1-month cumulative:             {model_adl.params['dgs1'] + model_adl.params['dgs1_lag1']:.4f}")
print(f"  2-month cumulative:             {model_adl.params['dgs1'] + model_adl.params['dgs1_lag1'] + model_adl.params['dgs1_lag2']:.4f}")
print(f"  R²: {model_adl.rsquared:.4f} (much higher than static model)")

# Check residual autocorrelation — should be near zero if well-specified
acf_adl = acf(model_adl.resid.dropna(), nlags=5)
print(f"  Residual ACF(1): {acf_adl[1]:.4f} (near zero → dynamics captured)")

Open empty Colab notebook →

Panel Data, Time Series Data, Causation

Panel data variance decomposition

Standard error comparison — why clustering matters

Pooled OLS vs fixed effects

Interest rates — levels vs changes

Autocorrelation — the smoking gun of non-stationarity

Spurious regression — R² that lies

ADL(2,2) model — dynamic multipliers

Python Libraries and Code