Regression, panel and spatial models

Three kinds of regression model for subnational data, what each one compares and when it is the right tool: least squares with fixed effects and clustered errors, panel estimators with the Hausman test, and spatial models with their diagnostics and impacts.

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab

A regression summarises how an outcome differs between observations that differ in something else. What separates the three kinds of model on this page is which observations are compared with which:

Model Function What it compares The right tool when
Least squares geo.regression Units with each other, in one year or in several pooled years; with fixed effects, only units of the same group You want a first measure of an association and to see how it moves as controls are added
Panel models geo.panel_model Units with each other (between) and each unit with itself over the years (within) The same units are observed in several years, and lasting differences between units may be behind the association
Spatial models geo.spatial_diagnostics, geo.spatial_model Units with each other, allowing neighbours to be related The residuals of a regression are similar among neighbours

All three describe associations. None of them, on these data, shows what causes what.

!pip install -q "geo4bolivia[panel]"
import numpy as np
import pandas as pd
import geo4bolivia as geo

geo.panel_model needs the linearmodels library, which the [panel] extra installs. The other functions work with the default install.

The data

The running example asks how the Sustainable Development Index (index_sdg, IDS, 0–100) of Bolivia’s 343 municipalities relates to how urban and how populous they are. geo.combine() puts the variables of several datasets in one panel; multidimensional poverty (sdg1_2_mpi, 0–1, higher is poorer) comes along for the last section.

df = geo.combine({"sdg": ["index_sdg", "population", "urbanization"],
                  "sdg01": ["sdg1_2_mpi"]}, scale="municipality")
df["log_population"] = np.log(df["population"])

variables = ["index_sdg", "urbanization", "log_population", "sdg1_2_mpi"]
columns = ["variable", "year", "count", "mean", "std", "min", "max"]
geo.describe(df, variables)[columns]
variable year count mean std min max
0 index_sdg 2012 343 50.123032 5.819148 37.200000 70.500000
1 index_sdg 2024 343 60.081341 5.263740 47.300000 77.300000
2 urbanization 2012 0 NaN NaN NaN NaN
3 urbanization 2024 343 26.580175 31.333405 0.000000 100.000000
4 log_population 2012 343 9.314551 1.134120 6.232448 14.183222
5 log_population 2024 343 9.449106 1.120815 6.774224 14.292354
6 sdg1_2_mpi 2012 343 0.234259 0.105528 0.018000 0.563000
7 sdg1_2_mpi 2024 343 0.063262 0.044054 0.003000 0.296000

Three things to keep in mind:

  • There are two periods, stored as 2012 and 2024.
  • Urbanization exists only for 2024, so any model that uses it is a cross-section of 2024.
  • Population is very unequal across municipalities, so the models use its logarithm, a column made here with numpy. One unit of log_population is a population 2.7 times larger; a doubling is 0.69. Tables and figures show a column of your own under its name, and the observatory’s variables under their labels.

Least squares: geo.regression

Several specifications side by side

A single regression says little about how solid an association is. The usual practice is to show several specifications (versions of the model) next to each other and watch what happens to the coefficient of interest as variables are added. In geo.regression, a list of lists is one model per list, and fixed_effects takes one entry per model.

res = geo.regression(
    df, "index_sdg",
    [["urbanization"],
     ["urbanization", "log_population"],
     ["urbanization", "log_population"]],
    fixed_effects=[None, None, "department"],
)
res.gt
Regression of Sustainable Development Index, IDS (0-100)
Ordinary least squares, 2024
Sustainable Development Index, IDS (0-100)
(1) (2) (3)
Urbanization rate (% of population) 0.094*** 0.086*** 0.098***
(0.008) (0.009) (0.009)
log_population 0.355 -0.004
(0.279) (0.291)
Intercept 57.582*** 54.436***
(0.313) (2.488)
Department fixed effects No No Yes
Observations 343 343 343
R² 0.313 0.317 0.428
Adjusted R² 0.311 0.313 0.411
*** p<0.01, ** p<0.05, * p<0.10. Standard errors in parentheses: heteroskedasticity-robust (HC1).

No year was given, so the function used the last year with data for every variable: 2024. Each column is a model; each cell is a coefficient with its standard error in parentheses below, and stars for the significance level.

  • Model (1): the coefficient on urbanization is 0.094. A municipality whose urbanization rate is 10 percentage points higher has, on average, an IDS 0.94 points higher. For scale, the standard deviation of the IDS in 2024 is 5.3 points. The intercept, 57.6, is the IDS the line gives to a municipality with no urban population. Urbanization alone accounts for 31% of the variation in the index (R² = 0.313).
  • Model (2) adds the size of the municipality. The coefficient on urbanization moves little, to 0.086, and the coefficient on log population (0.355, standard error 0.279) cannot be told apart from zero. Among municipalities with the same urbanization rate, larger ones do not have a detectably different IDS.
  • Model (3) adds fixed effects for the nine departments. Urbanization stays at 0.098 and R² rises to 0.428.

The coefficient of interest stays between 0.086 and 0.098 across the three models. A coefficient that moves a lot when a variable is added was partly picking up what that variable measures; this one does not depend on size or on the department.

The parts of the result are attributes: res.df is the tidy table of coefficients (model, term, estimate, std_error, p_value, conf_low, conf_high), res.summary has one row per model, res.fig draws the coefficients with their 95% confidence intervals, and res.models holds the fitted statsmodels objects.

res.summary
model n r2 r2_adj fixed_effects std_errors
0 1 343 0.313327 0.311313 none heteroskedasticity-robust (HC1)
1 2 343 0.316850 0.312831 none heteroskedasticity-robust (HC1)
2 3 343 0.428164 0.410940 department heteroskedasticity-robust (HC1)

Fixed effects: what “absorbing” means

A fixed effect for a grouping gives every group its own intercept. fixed_effects="department" adds one indicator variable (a dummy) per department, so each department has its own baseline level of the IDS. The same coefficients come out if you subtract from every variable the mean of its department and run the regression on what is left. That is why the group differences are said to be absorbed: everything that the municipalities of a department share, measured or not, is removed before the coefficients are estimated.

Two consequences:

  • The coefficient on urbanization in model (3) compares municipalities of the same department only. Whatever makes one department differ from another as a whole cannot be behind it.
  • A variable that is the same for all municipalities of a department cannot have a coefficient any more: the department intercepts have already taken all of its variation.

The table reports the fixed effects as a Yes/No row and leaves the eight dummy coefficients out, because they are not of interest. fixed_effects accepts a column of your table, the name of a coarser scale ("department"), "year", or a list of them.

Fixed effects do not make the estimate causal. They rule out one family of explanations (differences between departments) and leave the others: municipalities of the same department still differ in many ways that go with both urbanization and the index.

Clustered standard errors, and why nine clusters are too few

The standard errors above are heteroskedasticity-robust (HC1, the default): they allow the size of the errors to differ between municipalities, but treat the municipalities as independent of each other. Municipalities of the same department have things in common that the model leaves out, so their errors are probably related. Clustered standard errors allow any correlation between the errors of units in the same cluster and assume independence only between clusters. The coefficients do not change, only their uncertainty.

cluster= takes a column or a coarser scale. The chunk below fits model (3) with three kinds of standard error. Four provinces in different departments share the name Cercado; the package tells them apart, so cluster="province" gives 113 clusters, one per province.

x = ["urbanization", "log_population"]
kinds = {"robust (HC1)": None, "clustered by department": "department",
         "clustered by province": "province"}
fits = {kind: geo.regression(df, "index_sdg", x, fixed_effects="department",
                             cluster=cluster)
        for kind, cluster in kinds.items()}

columns = ["estimate", "std_error", "p_value", "conf_low", "conf_high"]
pd.DataFrame({kind: fit.df.set_index("term").loc["urbanization", columns]
              for kind, fit in fits.items()}).T.round(4)
estimate std_error p_value conf_low conf_high
robust (HC1) 0.0982 0.0094 0.0000 0.0797 0.1168
clustered by department 0.0982 0.0203 0.0013 0.0515 0.1450
clustered by province 0.0982 0.0102 0.0000 0.0780 0.1185

The estimate is 0.098 in the three rows. Clustering by department doubles the standard error, from 0.009 to 0.020, and widens the 95% interval from 0.08–0.12 to 0.05–0.15. The association is still clearly positive, but it is known much less precisely than the first table suggested.

Even that wider interval should not be trusted much. The result says why:

print(fits["clustered by department"].notes)
print(fits["clustered by department"].summary["std_errors"][0])
print(fits["clustered by province"].summary["std_errors"][0])
('Only 9 clusters: cluster-robust standard errors tend to be too small with few clusters',)
clustered by department (9 clusters)
clustered by province (113 clusters)

The formula behind clustered standard errors measures how the errors move together inside each cluster and then treats the clusters as the independent pieces of information. It is justified when there are many clusters. With nine departments, the uncertainty is itself estimated from nine pieces, and the resulting standard errors tend to be too small: intervals too narrow, and tests that reject too often. The function computes p-values and intervals with a t distribution with one degree of freedom less than the number of clusters (8 here), which helps but does not solve it, and it adds a note whenever there are fewer than 30 clusters. There is no sharp threshold.

What to do with few clusters:

  • Cluster at a finer level that still captures the dependence. With the 113 provinces as clusters the standard error is 0.010, close to the robust one. The formula is on firmer ground with that many clusters, but it assumes that municipalities of different provinces are unrelated, which neighbouring provinces may not be.
  • Model the dependence between neighbours directly, as the spatial models at the end of this page do.
  • Report the range. Here every version leaves the coefficient well away from zero, so the conclusion does not depend on the choice. When it does, say so.

Pooling the years

year= takes one year, a list of years, or "all", which stacks every year with data: each municipality then appears once per year. Urbanization has one year only, so this example uses population, which has both.

pooled = geo.regression(
    df, "index_sdg", [["log_population"], ["log_population"]], year="all",
    fixed_effects=[None, "year"], cluster="geo_id",
)
pooled.gt
Regression of Sustainable Development Index, IDS (0-100)
Ordinary least squares, 2012–2024 (2 years pooled)
Sustainable Development Index, IDS (0-100)
(1) (2)
log_population 2.366*** 2.110***
(0.232) (0.243)
Intercept 32.905***
(2.171)
Year fixed effects No Yes
Observations 686 686
R² 0.128 0.549
Adjusted R² 0.127 0.547
*** p<0.01, ** p<0.05, * p<0.10. Standard errors in parentheses: clustered by geo_id (343 clusters).

There are 686 observations: 343 municipalities in two periods.

  • Model (1) treats the 686 rows as one cloud of points. A municipality 2.7 times more populous has an IDS 2.4 points higher.
  • Model (2) adds a year fixed effect, an intercept for each period. The mean IDS rose from 50.1 to 60.1 between the two periods; the year effect absorbs that rise, so the coefficient (2.1) compares municipalities within the same period. R² jumps from 0.13 to 0.55, and almost all of that jump is the common rise, not population.

The coefficient on log population is about 2 here and was indistinguishable from zero in the 2024 table above. The difference is urbanization: larger municipalities are more urban (in 2024 the correlation between log population and the urbanization rate is 0.62), and once urbanization is in the model, size adds nothing. Urbanization is not available for 2012, so it cannot be added to the pooled model.

Two rows of the same municipality are not independent observations. cluster="geo_id" allows for that. If you pool without it, the result reminds you:

geo.regression(df, "index_sdg", "log_population", year="all").notes
("The years are pooled, so the same units appear several times; cluster='geo_id' allows for the dependence between a unit's rows",)

Pooled least squares mixes two comparisons: between municipalities, and between the two periods of the same municipality. Panel models separate them.

Panel models: geo.panel_model

A panel observes the same units in several years. geo.panel_model fits the four standard estimators side by side. They use the same data and differ in what they compare:

Estimator models= What it compares The question it answers
Pooled least squares "pooled" All rows, as one cloud A mix of the two below
Between "between" The units’ averages over the years, one row per unit Do units with a higher regressor have a higher outcome than other units?
Fixed effects (within) "fe" Each unit’s deviations from its own average When a unit’s regressor is above its own average, is its outcome too?
Random effects "re" A weighted mix of between and within The same as fixed effects, more precisely, if its assumption holds

The fixed-effects estimator gives every unit its own intercept. As with the department effects above, this absorbs every lasting difference between units, measured or not: whatever does not change over the years cannot be behind a within estimate. The price is that it uses only the variation over time, which is often small.

Random effects treat the unit intercepts as random draws that are unrelated to the regressors. When that holds, they use both sources of variation and are more precise. When it does not, they are pulled towards the between comparison, lasting differences included. The Hausman test checks it by comparing the fixed- and random-effects coefficients: if they differ by more than chance allows (a small p-value), the assumption fails and fixed effects are the estimates to read.

A long panel: night-time lights and population density

The SDG data have two periods. For a longer panel, geo.combine() joins night-time lights per capita (ln_NTLpc, in logs, 2012–2020) and population density (pop_density, people per km²). Both are computed on an earlier municipal map and cover 333 municipalities.

panel = geo.combine(["ntl", "pop"], scale="municipality")
d = geo.describe(panel, ["ln_NTLpc", "pop_density"])
d[d["year"].isin([2012, 2020])][["variable", "year", "count", "mean", "median"]]
variable year count mean median
11 ln_NTLpc 2012 318 13.098113 13.111378
19 ln_NTLpc 2020 328 13.909822 13.975348
31 pop_density 2012 333 52.071030 9.912329
39 pop_density 2020 333 62.842155 10.679666

Displaying the result shows the coefficients of the four estimators with their confidence intervals, and a reading of them:

pm = geo.panel_model(panel, "ln_NTLpc", "pop_density")
pm

Panel models of Log night-time lights per capita: 2,930 observations of 331 units over 9 years (2012–2020), estimated as pooled OLS, between, fixed effects, random effects.

  • Population density (people per km²), a one-unit higher value goes with Log night-time lights per capita that is:
    • 0.00117 higher pooling all rows (statistically significant at the 1% level);
    • 0.00121 higher between units (comparing the units’ averages over the years) (statistically significant at the 1% level);
    • 0.00217 higher within units (fixed effects: each unit compared with itself over the years) (statistically significant at the 1% level);
    • 0.0014 higher with random effects (a weighted mix of the between and within comparisons) (statistically significant at the 1% level).

The between estimate asks whether units with a higher regressor have a higher or lower outcome than other units; the within (fixed-effects) estimate asks whether a unit’s outcome moves when its own regressor moves, which leaves out every lasting difference between units. When the two disagree, the units that differ in the regressor also differ in other lasting ways that go with the outcome.

Hausman test: χ²(1) = 2.34, p = 0.126. The fixed- and random-effects estimates do not differ by more than chance allows, so random effects, the more precise of the two, are admissible. This is a lack of evidence against them, not a proof that they are right.

  • Note: Units with data in a single year (1 of 331) do not inform the fixed-effects estimates

These are associations, not causal effects. A causal reading needs a research design.

pm.gt
Panel models of Log night-time lights per capita
2,930 observations of 331 units, 2012–2020 (9 years)
Log night-time lights per capita
Pooled OLS Between Fixed effects Random effects
Population density (people per km²) 0.001*** 0.001*** 0.002*** 0.001***
(2.24e-04) (2.33e-04) (5.87e-04) (3.08e-04)
Intercept 13.518*** 13.472*** 13.464***
(0.067) (0.071) (0.070)
Unit effects No No Fixed Random
Year effects No No No No
Observations 2,930 331 2,930 2,930
Units 331 331 331 331
R² 0.047 0.052 0.006 0.004
R² within 0.005 0.005 0.006 0.005
R² between 0.051 0.052 0.018 0.051
*** p<0.01, ** p<0.05, * p<0.10. Standard errors in parentheses: clustered by geo_id (between: heteroskedasticity-robust).
Hausman test of fixed against random effects: χ²(1) = 2.34, p = 0.1258

The model uses the rows with both variables: 2,930 observations of 331 municipalities over nine years.

  • Between (0.0012): municipalities that are denser on average over the nine years have more light per person. A difference of 100 people per km² goes with 0.12 more log points of lights per capita, about 13% more light per person.
  • Within (0.0022): in the years when a municipality is denser than its own average, it also has more light per person than its own average.
  • The Hausman test (χ² = 2.34 with 1 degree of freedom, p = 0.13) does not reject random effects, whose estimate (0.0014) lies between the other two.
  • Density accounts for very little of the variation in either direction: the between R² is 0.05 and the within R² is below 0.01.

Standard errors are clustered by municipality by default (cluster=True), which allows the errors of the same municipality to be related across years. .summary and .hausman hold the numbers:

print(pm.hausman)
pm.summary[["estimator", "n", "units", "years", "r2_within", "r2_between"]]
Hausman(statistic=2.343637731930409, df=1, p_value=0.12579548799877696)
estimator n units years r2_within r2_between
0 Pooled OLS 2930 331 9 0.004524 0.050703
1 Between 331 331 9 0.004609 0.051903
2 Fixed effects 2930 331 9 0.005731 0.018331
3 Random effects 2930 331 9 0.005018 0.050554

Year effects change the within estimate

Between 2012 and 2020 both variables rose on average: the table above shows the mean of log lights per capita going from 13.1 to 13.9, and mean density from 52 to 63 people per km². A within estimator compares each municipality with itself over the years, so a rise that all municipalities share counts as association, even if the two rises have nothing to do with each other. time_effects=True adds an intercept for each year, which removes whatever is common to all units in a year.

pm_t = geo.panel_model(panel, "ln_NTLpc", "pop_density", time_effects=True)
pm_t.gt
Panel models of Log night-time lights per capita
2,930 observations of 331 units, 2012–2020 (9 years)
Log night-time lights per capita
Pooled OLS Between Fixed effects Random effects
Population density (people per km²) 0.001*** 0.001*** -0.002*** 3.39e-04***
(2.21e-04) (2.33e-04) (5.74e-04) (1.15e-04)
Intercept 13.035*** 13.472*** 12.990***
(0.078) (0.071) (0.081)
Unit effects No No Fixed Random
Year effects Yes No Yes Yes
Observations 2,930 331 2,930 2,930
Units 331 331 331 331
R² 0.097 0.052 0.008 0.419
R² within 0.447 0.005 -0.014 0.456
R² between 0.033 0.052 -0.019 0.005
*** p<0.01, ** p<0.05, * p<0.10. Standard errors in parentheses: clustered by geo_id (between: heteroskedasticity-robust).
Hausman test of fixed against random effects: χ²(1) = 38.71, p < 0.0001

The within estimate changes sign, from 0.0022 to −0.0019. The question it now answers is narrower: in the years when a municipality’s density is higher than its own average by more than the country’s movement that year, are its lights per capita higher too? The answer is no: they are lower. The between estimate is the same as before, because averaging over the years leaves no year to adjust for.

Two cautions about this result:

  • The negative within estimate is partly built in. Lights per capita are lights divided by population, and density is population divided by an area that does not change. Within a municipality, density rises only when population rises, and if lights do not keep pace, lights per capita fall. A variable that sits in the numerator of one side and the denominator of the other produces a negative association by construction.
  • The Hausman test now rejects random effects (χ² = 38.7, p < 0.001), where it did not before. The test is about a specification, not about the data in general. Decide whether the model needs year effects first, on the grounds of what could be common to all units, and read the test after.

Two periods: the SDG data

With two periods the same issue is sharper. Here are the four estimators for the IDS and log population:

two = geo.panel_model(df, "index_sdg", "log_population")
two.gt
Panel models of Sustainable Development Index, IDS (0-100)
686 observations of 343 units, 2012–2024 (2 years)
Sustainable Development Index, IDS (0-100)
Pooled OLS Between Fixed effects Random effects
log_population 2.366*** 2.135*** 19.390*** 2.393***
(0.232) (0.244) (2.692) (0.231)
Intercept 32.905*** 35.070*** 32.655***
(2.169) (2.264) (2.161)
Unit effects No No Fixed Random
Year effects No No No No
Observations 686 343 686 686
Units 343 343 343 343
R² 0.128 0.200 0.238 0.124
R² within 0.054 0.049 0.238 0.055
R² between 0.198 0.200 -4.376 0.197
*** p<0.01, ** p<0.05, * p<0.10. Standard errors in parentheses: clustered by geo_id (between: heteroskedasticity-robust).
Hausman test of fixed against random effects: χ²(1) = 83.16, p < 0.0001

Between municipalities, a population 2.7 times larger goes with an IDS 2.1 points higher. The fixed-effects estimate is 19.4, nine times that. It is not a finding about population.

With two periods, the fixed-effects estimate is the regression of each unit’s change in the outcome on its change in the regressor. Without year effects, that regression has no intercept, so the line is forced through the origin. geo.change() shows what the changes look like:

chg_ids = geo.change(df, "index_sdg")["index_sdg_change"]
chg_pop = geo.change(df, "log_population")["log_population_change"]
print("IDS rose in", (chg_ids > 0).sum(), "of", len(chg_ids), "municipalities;",
      "mean change", round(chg_ids.mean(), 1))
print("population grew in", (chg_pop > 0).sum(), "; mean change of the log",
      round(chg_pop.mean(), 2))
IDS rose in 342 of 343 municipalities; mean change 10.0
population grew in 260 ; mean change of the log 0.13

The index rose by about 10 points almost everywhere, and population grew by a small amount in most places. A line through the origin can only account for a rise of 10 points that goes with a rise of 0.13 in log population by being very steep. The estimator attributes a change common to all municipalities to the regressor.

two_t = geo.panel_model(df, "index_sdg", "log_population", time_effects=True)
two_t.gt
Panel models of Sustainable Development Index, IDS (0-100)
686 observations of 343 units, 2012–2024 (2 years)
Sustainable Development Index, IDS (0-100)
Pooled OLS Between Fixed effects Random effects
log_population 2.110*** 2.135*** -0.416 1.888***
(0.243) (0.244) (0.948) (0.250)
Intercept 30.468*** 35.070*** 32.539***
(2.235) (2.264) (2.288)
Unit effects No No Fixed Random
Year effects Yes No Yes Yes
Observations 686 343 686 686
Units 343 343 343 343
R² 0.549 0.200 0.001 0.854
R² within 0.918 0.049 -0.010 0.919
R² between 0.200 0.200 -0.146 0.198
*** p<0.01, ** p<0.05, * p<0.10. Standard errors in parentheses: clustered by geo_id (between: heteroskedasticity-robust).
Hausman test of fixed against random effects: χ²(1) = 11.82, p = 0.0006

With time_effects=True the common rise goes to the year effect (the intercept of the regression of changes), and the fixed-effects estimate becomes −0.4 with a standard error of 0.9: municipalities whose population grew more did not see their IDS rise more, or less, than the others. The between association (2.1) remains: larger municipalities have a higher index, but growing did not go with improving faster.

.interpret() warns about this whenever a fixed-effects model has two years and no year effects. With two periods, use time_effects=True unless you have a reason to believe that nothing changed for all units at once.

Spatial models

The Lagrange multiplier tests and the model they point to

The other rows are Lagrange multiplier (LM) tests. Each one asks, using only the least-squares fit, whether adding one kind of spatial term would improve the model:

  • LM lag: a term for the neighbours’ outcome. The alternative is the spatial lag model, in which the IDS of a municipality is related to the IDS of its neighbours.
  • LM error: a term for the neighbours’ errors. The alternative is the spatial error model, in which what the model leaves out is spatially correlated but the outcome does not depend on the neighbours’ outcome.

The two simple tests react to either kind of dependence, so both usually reject together, as here (168.5 and 187.1). The robust versions test one kind while allowing for the other, and they decide:

print(diag.recommendation)
print(diag.reasoning)
error
Both robust LM tests reject at alpha = 0.05 (robust LM lag = 4.50, robust LM error = 23.08), so the larger robust statistic wins and points to the error model. With both channels active, consider durbin (SDM) - it nests both the lag and the spatially-lagged-X channels and lets the data separate them.

The rule is: if neither simple test rejects, keep least squares; otherwise read the robust tests, and choose the model whose robust test rejects, or the one with the larger statistic when both do. (If neither robust test rejects, the larger simple statistic gives a tentative answer.) Here the robust LM error (23.1) is much larger than the robust LM lag (4.5), so the tests point to the error model. The robust lag test also rejects at 5% (p = 0.03), which is why the function adds that a Durbin model is worth a look.

Lag, error and Durbin models

Writing \(W y\) for the average outcome of a unit’s neighbours and \(W x\) for the average of their regressor, the three main models are:

Model model= Equation What is spatial
Spatial lag "lag" \(y = \rho W y + \beta x + \varepsilon\) The outcome depends on the neighbours’ outcome
Spatial error "error" \(y = \beta x + u,\; u = \lambda W u + \varepsilon\) The unexplained part is shared with neighbours
Spatial Durbin "durbin" \(y = \rho W y + \beta x + \theta W x + \varepsilon\) The neighbours’ outcome and the neighbours’ regressor

The Durbin model contains the lag model (when θ = 0), and it can also reproduce the error model, so it is the cautious choice when the tests do not separate the two. geo.spatial_model also fits "slx" (the neighbours’ regressor only), "durbin_error" and "ols", the last one for comparison.

kinds = ["ols", "lag", "error", "durbin"]
sp = {kind: geo.spatial_model(df, "index_sdg", "urbanization", model=kind)
      for kind in kinds}
coef = lambda fit: fit.df.set_index("variable").loc["urbanization", "estimate"]
pd.DataFrame({kind: {"coefficient on urbanization": coef(fit),
                     "rho": fit.rho, "lambda": fit.lam, "AIC": fit.aic,
                     "pseudo R²": fit.r2}
              for kind, fit in sp.items()}).T.round(3)
coefficient on urbanization rho lambda AIC pseudo R²
ols 0.094 NaN NaN 1986.795 0.313
lag 0.072 0.618 NaN 1854.558 0.579
error 0.081 NaN 0.675 1849.412 0.313
durbin 0.080 0.668 NaN 1848.997 0.598
  • The spatial parameters are large: ρ = 0.62 in the lag model, λ = 0.67 in the error model. Neighbouring municipalities resemble each other well beyond what urbanization accounts for.
  • AIC (lower is better, on the same data) falls from 1,987 for least squares to about 1,850 for the spatial models. The error and Durbin models are practically tied (1,849.4 and 1,849.0) and both are ahead of the lag model (1,854.6), in line with the tests.
  • Compare these models by AIC, not by .r2. It is a pseudo-R² that is not comparable between them: the error model’s is the same 0.313 as least squares, because it counts only what the regressor accounts for.

The error model, which the tests recommend, is the simplest to read. Its coefficient has the same meaning as in least squares:

sp["error"]
Spatial error (sem) model of Sustainable Development Index, IDS (0-100)
ML estimates | n = 343 | R² = 0.313 | AIC = 1849.4
Estimate Std. Error z p
CONSTANT 57.9036*** (0.5841) 99.14 0.0000
Urbanization rate (% of population) 0.0815*** (0.0066) 12.26 0.0000
lambda 0.6749*** (0.0488) 13.82 0.0000
*** p<0.01, ** p<0.05, * p<0.10 (z-based)
W: queen contiguity (1 island(s) attached to nearest neighbor), row-standardized, n=343

A spatial error (SEM) model of Sustainable Development Index, IDS (0-100) was estimated by ML on 343 units (weights: queen contiguity (1 island(s) attached to nearest neighbor), row-standardized, n=343).

The spatial error parameter is λ = 0.675 (statistically significant at the 1% level): unmodeled shocks are spatially correlated across neighbors, which matters for standard errors more than for the coefficients themselves.

This specification has no direct/indirect impact decomposition; coefficients are read directly from the table.

Model fit: (pseudo-)R² = 0.313, AIC = 1849 (compare AIC across specifications estimated on the same sample).

These are associations, not causal effects. A causal reading needs a research design.

A municipality 10 points more urban has an IDS about 0.8 points higher, against 0.94 in least squares, and the standard error now allows for the dependence between neighbours.

Direct, indirect and total impacts

In a lag or Durbin model a coefficient is not the whole association. The outcome of each unit is tied to its neighbours’ outcomes, which are tied to their own regressors and to their neighbours, and so on back to the unit itself. The model therefore implies three quantities for each regressor, called impacts:

  • Direct: how the outcome of a unit differs with its own regressor, including the part that comes back through its neighbours.
  • Indirect (often called spillover): how the outcome of a unit differs with the regressor of the other units, summed over them.
  • Total: the sum of the two.
impacts = {kind: sp[kind].impacts.drop(columns="term") for kind in ["lag", "durbin"]}
pd.concat(impacts).droplevel(1).round(3)
variable direct se_direct indirect se_indirect total se_total
lag urbanization 0.080 0.007 0.109 0.02 0.189 0.023
durbin urbanization 0.091 0.008 0.051 0.03 0.142 0.030

In the lag model the coefficient on urbanization is 0.072, but the total impact is 0.189, more than twice as much: it equals β / (1 − ρ) = 0.072 / (1 − 0.618). Reading 0.072 as “the association with urbanization” would understate what the model says.

The two models disagree about the indirect part. The lag model puts it at 0.109, larger than the direct impact. This is a property of the model as much as of the data: with a single ρ, the ratio of indirect to direct impact is the same for every regressor and is set by ρ and the weights alone. The Durbin model lets the neighbours’ urbanization have its own coefficient, and gives an indirect impact of 0.051 with a standard error of 0.030, less than twice its standard error. The error model has no indirect impact by construction.

The standard errors of the impacts come from simulation (n_draws=10_000 draws by default; seed= makes them reproducible).

What can be said from the three models together:

  • The direct association is stable: between 0.08 and 0.09 IDS points per percentage point of urbanization in every model.
  • Whether a municipality’s index also goes with the urbanization of its neighbours is not settled by these data: the answer runs from zero (error model) through an imprecise 0.05 (Durbin) to 0.11 (lag), and the tests and the AIC favour the models that give the smaller answers.
  • “Spillover” is the name of a term in the model. It measures an association with the neighbours’ values, and it does not show that urbanization in one municipality changes anything in another.

The results also depend on who counts as a neighbour. A quick check with other weights, the six nearest municipalities instead of those sharing a border:

w6 = geo.weights("municipality", method="knn", k=6)
diag6 = geo.spatial_diagnostics(df, "index_sdg", "urbanization", w=w6)
error6 = geo.spatial_model(df, "index_sdg", "urbanization", model="error", w=w6)
print(diag6.recommendation, "| coefficient:", round(coef(error6), 3),
      "| lambda:", round(error6.lam, 2))
error | coefficient: 0.082 | lambda: 0.72

The tests point to the error model again, and its coefficient (0.083) is close to the 0.081 found with shared borders.

Another outcome, another model

The tests do not always point the same way. For multidimensional poverty on urbanization they choose the lag model:

diag_mpi = geo.spatial_diagnostics(df, "sdg1_2_mpi", "urbanization")
print(diag_mpi.recommendation)
diag_mpi.df.round(4)
lag
test statistic df p
0 moran_residuals 0.3504 NaN 0.0000
1 lm_lag 111.6706 1.0 0.0000
2 lm_error 104.0186 1.0 0.0000
3 robust_lm_lag 9.1127 1.0 0.0025
4 robust_lm_error 1.4607 1.0 0.2268
5 lm_sarma 113.1313 2.0 0.0000

Here the robust LM lag rejects (9.1, p = 0.003) and the robust LM error does not (1.5, p = 0.23).

lag_mpi = geo.spatial_model(df, "sdg1_2_mpi", "urbanization", model="lag")
lag_mpi
Spatial lag (sar) model of Multidimensional poverty index (0-1)
ML estimates | n = 343 | R² = 0.453 | AIC = -1338.5
Estimate Std. Error z p
CONSTANT 0.0378*** (0.0043) 8.82 0.0000
Urbanization rate (% of population) -0.0005*** (0.0001) -7.76 0.0000
W_Multidimensional poverty index (0-1) 0.5903*** (0.0512) 11.52 0.0000
*** p<0.01, ** p<0.05, * p<0.10 (z-based)
W: queen contiguity (1 island(s) attached to nearest neighbor), row-standardized, n=343

A spatial lag (SAR) model of Multidimensional poverty index (0-1) was estimated by ML on 343 units (weights: queen contiguity (1 island(s) attached to nearest neighbor), row-standardized, n=343).

The spatial autoregressive parameter is ρ = 0.59 (statistically significant at the 1% level): outcomes move together with neighbors’ outcomes, so any local difference is amplified through the neighborhood multiplier 1/(1-ρ) ≈ 2.44 rather than staying put.

For Urbanization rate (% of population), a one-unit higher value is associated with a direct change of -0.0005 in the same unit’s Multidimensional poverty index (0-1) and an indirect (spillover) change of -0.000616 summed across its neighbors — a total association of -0.00112 (statistically significant at the 1% level).

Model fit: (pseudo-)R² = 0.453, AIC = -1339 (compare AIC across specifications estimated on the same sample).

These are associations, not causal effects. A causal reading needs a research design.

lag_mpi.impacts.round(5)
variable term direct se_direct indirect se_indirect total se_total
0 urbanization Urbanization rate (% of population) -0.0005 0.00006 -0.00062 0.00014 -0.00112 0.00018

Poverty is lower where urbanization is higher, and the poverty of neighbouring municipalities moves together (ρ = 0.59). A municipality 10 points more urban has a poverty index 0.005 lower directly and 0.011 lower in total, against a mean of 0.063 in 2024. As before, the indirect part is what a lag model implies once ρ is estimated, and it should be read with the same caution.

Both functions take fixed_effects= (a column or a coarser scale, added as dummies), w= for other weights, and year=; they analyse one year at a time. geo.spatial_model estimates by maximum likelihood by default (method="ml") and by the generalized method of moments with method="gm".

In short

  • geo.regression puts specifications side by side. The IDS is 0.09 to 0.10 points higher per percentage point of urbanization in 2024, with or without population and department fixed effects.
  • Fixed effects absorb everything the units of a group share, so the coefficients compare units of the same group only. They rule out one family of explanations, not all.
  • Clustered standard errors allow for related errors inside a cluster. With nine departments they are unreliable and tend to be too small; cluster at a finer level, model the spatial dependence, or report the range.
  • year="all" pools the years; add "year" fixed effects for what changed everywhere and cluster="geo_id" for the repeated units.
  • geo.panel_model separates the between comparison (units with each other) from the within comparison (each unit with itself). The Hausman test says whether random effects are admissible for a given specification.
  • Without year effects, a within estimate also absorbs changes common to all units. It turned a null association between population growth and the change in the IDS into a coefficient of 19, and it changed the sign of the lights and density estimate. With two periods, use time_effects=True.
  • geo.spatial_diagnostics tests the residuals of least squares for spatial dependence and points to a lag or an error model; geo.spatial_model fits it. In lag and Durbin models, read the direct, indirect and total impacts, not the coefficients.
  • Every number on this page is an association between municipalities. None of the models identifies a cause.

Where next