# geo4bolivia > Spatial data science for Bolivia's subnational development data: find, load, map and analyse it across spatial scales. geo4bolivia is the Python package of the GeoDevelopment Observatory of Bolivia. It reads the observatory's published data (https://github.com/quarcs-lab/gdo4bolivia) and runs the fundamental analyses of spatial data science on them. Install with `pip install geo4bolivia` and `import geo4bolivia as geo`. Every analysis function takes a dataset id or a panel with `geo_id` and `year`, finds boundaries, weights and labels itself, and returns a result object with `.df`, `.fig` (Plotly), `.gt` (table) and `.interpret()`. ## Pages - [Analyze](https://quarcs-lab.github.io/geo4bolivia/analyze.html): A tour of the analysis functions on one question: did sustainable development become more equal across Bolivia's municipalities, and what role does space play? - [Changelog](https://quarcs-lab.github.io/geo4bolivia/changelog.html): What changed in each release of geo4bolivia. - [Explore](https://quarcs-lab.github.io/geo4bolivia/explore.html): A first pass over one indicator, the Sustainable Development Index: summaries, distributions, extremes, correlations, trends and maps, before any model. - [Get started](https://quarcs-lab.github.io/geo4bolivia/get-started.html): Install geo4bolivia, find a variable, load it, map it and run a first spatial analysis, in ten minutes. - [For AI assistants and LLMs](https://quarcs-lab.github.io/geo4bolivia/use-with-llms.html): How an AI assistant should install and call geo4bolivia, and the files that describe the package to it. - [Case study: the Sustainable Development Index, ~2012 to ~2024](https://quarcs-lab.github.io/geo4bolivia/articles/case-study-sdg.html): One complete analysis with geo4bolivia: how the Sustainable Development Index of Bolivia's 343 municipalities changed between two periods, which goals moved, whether municipalities converged, and where. - [Convergence: β, σ and clubs](https://quarcs-lab.github.io/geo4bolivia/articles/convergence.html): Three questions about catching up, and the method for each: do units that start lower grow faster, is dispersion falling, and do units head for one path or for several? - [The data model: scales, geo IDs, datasets and metadata](https://quarcs-lab.github.io/geo4bolivia/articles/data-model.html): How the observatory's data are organised (spatial scales, geo IDs, panels and metadata) and how the package reads them. - [Exploring the data](https://quarcs-lab.github.io/geo4bolivia/articles/exploring.html): A question-driven exploration of multidimensional poverty in Bolivia's municipalities, and the pitfalls of these data that a newcomer should know. - [Spatial heterogeneity: MGWR](https://quarcs-lab.github.io/geo4bolivia/articles/heterogeneity.html): Multiscale geographically weighted regression asks whether an association differs from place to place and at what spatial scale it operates: how to run it with geo.mgwr and how to read its bandwidths, coefficient maps and diagnostics. - [Articles](https://quarcs-lab.github.io/geo4bolivia/articles/index.html): One article per method, written around Bolivia's data, and a complete case study. - [Regional inequality](https://quarcs-lab.github.io/geo4bolivia/articles/inequality.html): How unequal Bolivia's municipalities are, whether that is changing, and whether the inequality lies between departments or inside them: Gini, Theil, the Lorenz curve and Theil's decomposition. - [Regression, panel and spatial models](https://quarcs-lab.github.io/geo4bolivia/articles/models.html): Three kinds of regression model for subnational data, what each one compares and when it is the right tool: least squares with fixed effects and clustered errors, panel estimators with the Hausman test, and spatial models with their diagnostics and impacts. - [Building your own indicators](https://quarcs-lab.github.io/geo4bolivia/articles/own-indicators.html): From a table or a raster of your own to an analysis: link it to the geo IDs, compute zonal statistics, move it across scales, save it and analyse it. - [Maps and spatial dependence](https://quarcs-lab.github.io/geo4bolivia/articles/spatial-dependence.html): From a map to a test: classed maps, spatial weights, global Moran's I and LISA clusters of multidimensional poverty in Bolivia's municipalities, and how far the conclusions depend on the choices made. - [Transitions and mobility](https://quarcs-lab.github.io/geo4bolivia/articles/transitions.html): How units move inside a distribution: changes of rank between two years, moves between spatial cluster types, and Markov transition matrices with and without the neighbours. - [Function reference](https://quarcs-lab.github.io/geo4bolivia/reference/index.html): every public function, with an example - [llms-full.txt](https://quarcs-lab.github.io/geo4bolivia/llms-full.txt): all of the above in one file ## Find and understand the data - [`geo.datasets(ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/datasets.html): The catalog: one row per dataset and scale. - [`geo.search(text: 'str', dataset: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/search.html): Find variables whose name, label or description contain every word of ``text``. - [`geo.info(dataset: 'str', ref: 'str | None' = None) -> 'DatasetInfo'`](https://quarcs-lab.github.io/geo4bolivia/reference/info.html): What a dataset is: description, coverage (scales, units, years), actual reference - [`geo.dictionary(dataset: 'str', scale: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/dictionary.html): The variable dictionary of a dataset, one row per column. - [`geo.reference_years(dataset: 'str', variable: 'str | None' = None, scale: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/reference_years.html): The actual years behind each stored year, one row per variable and stored year. - [`geo.cite(dataset: 'str', style: 'str' = 'apa', ref: 'str | None' = None) -> 'str'`](https://quarcs-lab.github.io/geo4bolivia/reference/cite.html): How to cite a dataset: its recommended citation, then the credits its sources require. - [`geo.labels(data, own: 'dict[str, str] | None' = None, ref: 'str | None' = None) -> 'dict[str, str]'`](https://quarcs-lab.github.io/geo4bolivia/reference/labels.html): Readable labels for the columns of a panel (or every field of a dataset id). - [`geo.describe(df: 'pd.DataFrame', variables=None, by: 'str | None' = 'year', ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/describe.html): Summary statistics of a panel's variables, per ``by`` group (default: per year). ## Load data, units and boundaries - [`geo.load(dataset: 'str', scale: 'str | None' = None, years=None, variables=None, wide: 'str | None' = None, geometry: 'bool' = False, names: 'bool' = False, groups=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/load.html): Load a dataset as a panel: one row per unit (``geo_id``) and year. - [`geo.add_names(df: 'pd.DataFrame', groups=None, scale: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/add_names.html): Add the unit ``name`` and, for each scale in ``groups``, the name of the unit it lies in. - [`geo.units(scale: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/units.html): All units (or the units of one scale) with names, parents and source keys. - [`geo.boundaries(scale: 'str', ref: 'str | None' = None) -> 'gpd.GeoDataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/boundaries.html): Unit polygons for a scale (CRS84), with ``geo_id`` and ``name``. - [`geo.crosswalk(source: 'str', target: 'str', ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/crosswalk.html): Links between two scales: ``geo_id_from``, ``geo_id_to``, ``weight_area``. - [`geo.scales(active_only: 'bool' = False, ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/scales.html): The registry of spatial scales as a table, one row per scale (``geo.scales()``). - [`geo.ancestors(scale: 'str', ref: 'str | None' = None) -> 'list[str]'`](https://quarcs-lab.github.io/geo4bolivia/reference/ancestors.html): Coarser scales a unit of ``scale`` nests in, in breadth-first order. - [`geo.scale_of(geo_id: 'str', ref: 'str | None' = None) -> 'str'`](https://quarcs-lab.github.io/geo4bolivia/reference/scale_of.html): The scale a geo ID belongs to, from the ID patterns (e.g. a municipality ID -> ``"municipality"``). - [`geo.config() -> 'dict'`](https://quarcs-lab.github.io/geo4bolivia/reference/config.html): The project settings: the package's project.yaml, with the data repository's on top inside a checkout. ## Work with panels and scales - [`geo.change(df: 'pd.DataFrame', variables=None, first: 'int | None' = None, last: 'int | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/change.html): The change of each unit between two years of a panel, one row per unit. - [`geo.aggregate(df: 'pd.DataFrame', to, dataset: 'str | None' = None, rules: 'dict | None' = None, variables=None, weights: 'pd.DataFrame | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/aggregate.html): Aggregate a panel up to a coarser, nested scale (or to several at once). - [`geo.combine(datasets, scale: 'str', years=None, extra=(), rules: 'dict | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/combine.html): Merge datasets into one panel at ``scale``, aggregating finer ones up. - [`geo.overlay(df: 'pd.DataFrame', target, extensive=(), intensive=(), source=None, id_col: 'str' = 'geo_id') -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/overlay.html): Area-weighted interpolation of a panel onto non-nested units. - [`geo.overlay_weights(source, target, id_col: 'str' = 'geo_id') -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/overlay_weights.html): Area of every source-target intersection, in an equal-area projection. ## Build your own indicators - [`geo.link(df: 'pd.DataFrame', scale: 'str', on: 'str', key: 'str' = 'auto', errors: 'str' = 'raise', ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/link.html): Attach ``geo_id`` to a table that identifies units by name or by a source code. - [`geo.check(df: 'pd.DataFrame', scale: 'str | None' = None, time: 'str | None' = 'year', ref: 'str | None' = None) -> 'list[str]'`](https://quarcs-lab.github.io/geo4bolivia/reference/check.html): Problems that would stop ``df`` from being a valid panel at ``scale`` (empty list = OK). - [`geo.zonal(raster, scale, stats=('sum', 'mean'), name: 'str' = 'value', year: 'int | None' = None, band: 'int' = 1, all_touched: 'bool' = False, ref: 'str | None' = None) -> 'pd.DataFrame'`](https://quarcs-lab.github.io/geo4bolivia/reference/zonal.html): Summarise a raster (GeoTIFF path or URL) over the units of a scale. - [`geo.save(df: 'pd.DataFrame', name: 'str', scale: 'str', directory='.', title: 'str | None' = None, description: 'str' = '', variables: 'dict | None' = None, sources=(), citation: 'str | None' = None, license: 'str' = 'CC-BY-4.0') -> 'dict'`](https://quarcs-lab.github.io/geo4bolivia/reference/save.html): Save a panel exactly like the published datasets, ready to share or contribute. ## Explore - [`geo.correlation(data, variables=None, year=None, *, method: 'str' = 'pearson', digits: 'int' = 2, title: 'str | None' = None, ref: 'str | None' = None) -> 'CorrelationResult'`](https://quarcs-lab.github.io/geo4bolivia/reference/correlation.html): Correlations between the variables of a panel in one year. - [`geo.extremes(data, variable: 'str', year=None, *, n: 'int' = 5, title: 'str | None' = None, ref: 'str | None' = None) -> 'ExtremesResult'`](https://quarcs-lab.github.io/geo4bolivia/reference/extremes.html): The units with the highest and the lowest values of a variable in one year. - [`geo.distribution(data, variable: 'str', year=None, *, by: 'str | None' = None, kind: 'str' = 'histogram', bins: 'int' = 30, title: 'str | None' = None, ref: 'str | None' = None) -> 'DistributionResult'`](https://quarcs-lab.github.io/geo4bolivia/reference/distribution.html): The distribution of a variable across units, per year or per group. - [`geo.scatter(data, x: 'str', y: 'str', year=None, *, color: 'str | None' = None, size: 'str | None' = None, fit: 'str | None' = 'ols', log_x: 'bool' = False, log_y: 'bool' = False, title: 'str | None' = None, ref: 'str | None' = None) -> 'ScatterResult'`](https://quarcs-lab.github.io/geo4bolivia/reference/scatter.html): Two variables against each other in one year, one point per unit. - [`geo.trend(data, variable: 'str', *, by: 'str | None' = None, units: 'bool' = True, stat: 'str' = 'mean', max_units: 'int' = 400, title: 'str | None' = None, ref: 'str | None' = None) -> 'TrendResult'`](https://quarcs-lab.github.io/geo4bolivia/reference/trend.html): A variable over time: its mean (or median) across units, with a band, and each unit's path. ## Maps and spatial dependence - [`geo.map(data, variable: 'str', year=None, scale: 'str | None' = None, title: 'str | None' = None, simplify: 'float | str | None' = 'auto', ref: 'str | None' = None, *, scheme: 'str | None' = None, k: 'int' = 5, bins=None, change: 'bool' = False, **kwargs)`](https://quarcs-lab.github.io/geo4bolivia/reference/map.html): An interactive choropleth map (Plotly) of one variable: in one year, in several years side by - [`geo.weights(data, method: 'str' = 'queen', *, k: 'int' = 6, threshold: 'float | None' = None, power: 'float' = 1.0, row_standardize: 'bool' = True, attach_islands: 'bool' = True, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/weights.html): Spatial weights for the units of a scale, a dataset or a panel. - [`geo.connectivity(data, w=None, *, method: 'str' = 'queen', tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/connectivity.html): A map of the neighbour links of a set of weights, with the distribution of neighbours per unit. - [`geo.moran(data, variable: 'str', year=None, *, w=None, permutations: 'int' = 999, seed: 'int | None' = 12345, title: 'str | None' = None, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/moran.html): Global Moran's I of a variable: is it clustered in space? - [`geo.lisa(data, variable: 'str', year=None, *, w=None, permutations: 'int' = 999, seed: 'int | None' = 12345, alpha: 'float' = 0.05, panels: 'str' = 'both', tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/lisa.html): Local Moran (LISA) clusters of a variable: where are the hot spots, cold spots and outliers? ## Convergence - [`geo.beta_convergence(data, variable: 'str', controls=None, *, first=None, last=None, model: 'str' = 'ols', w=None, fixed_effects=None, vcov: 'str' = 'hetero', draws: 'int' = 10000, seed: 'int' = 12345, min_obs: 'int' = 10, title: 'str | None' = None, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/beta_convergence.html): Beta-convergence: did the units that started lower grow faster between two years? - [`geo.sigma_convergence(data, variable: 'str', *, first=None, last=None, vcov: 'str' = 'hetero', title: 'str | None' = None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/sigma_convergence.html): Sigma-convergence: does the dispersion of a variable across units fall over time? - [`geo.convergence_clubs(data, variable: 'str', *, first=None, last=None, log: 'bool' = False, hp_filter: 'bool' = True, hp_lambda: 'float' = 400.0, trim: 'float' = 0.3, tcrit: 'float' = -1.65, cr: 'float' = 0.0, increment: 'float' = 0.05, max_cr: 'float' = 3.0, fraction: 'float' = 0.0, adjust: 'bool' = False, merge: 'str' = 'ps', map: 'bool' = True, tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/convergence_clubs.html): Convergence clubs (Phillips and Sul): do all units converge, or do groups of them? ## Transitions and mobility - [`geo.transitions(data, variable: 'str', *, k: 'int' = 5, scheme: 'str' = 'quantiles', bins=None, per_year: 'bool' = True, relative: 'bool' = False, title: 'str | None' = None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/transitions.html): Markov transitions between classes of a variable's distribution: how mobile are the units? - [`geo.spatial_transitions(data, variable: 'str', *, w=None, k: 'int' = 5, m: 'int | None' = None, per_year: 'bool' = False, relative: 'bool' = True, title: 'str | None' = None, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/spatial_transitions.html): Spatial Markov transitions: does a unit's mobility depend on where its neighbours stand? - [`geo.lisa_transitions(data, variable: 'str', first=None, last=None, *, w=None, permutations: 'int' = 999, seed: 'int | None' = 12345, alpha: 'float' = 0.05, tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/lisa_transitions.html): Transitions between local Moran (LISA) types: did the hot spots and cold spots persist? - [`geo.rank_mobility(data, variable: 'str', first=None, last=None, *, k: 'int' = 5, top: 'int' = 10, ascending: 'bool' = False, title: 'str | None' = None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/rank_mobility.html): Rank mobility between two years: how much did the units change position? ## Inequality - [`geo.inequality(data, variable: 'str', *, measures=('gini', 'theil'), first=None, last=None, spatial: 'bool' = False, w=None, permutations: 'int' = 99, lorenz: 'bool' = True, title: 'str | None' = None, gdf=None, ref: 'str | None' = None) -> 'InequalityResult'`](https://quarcs-lab.github.io/geo4bolivia/reference/inequality.html): Inequality between units in each year: is it narrowing or widening? - [`geo.theil_decomposition(data, variable: 'str', by: 'str | None' = None, *, permutations: 'int' = 0, seed: 'int' = 12345, title: 'str | None' = None, ref: 'str | None' = None) -> 'TheilResult'`](https://quarcs-lab.github.io/geo4bolivia/reference/theil_decomposition.html): Theil's index split in two: inequality between groups of units and inequality within them. ## Models - [`geo.regression(data, y: 'str', x, year=None, *, fixed_effects=None, cluster: 'str | None' = None, robust: 'bool' = True, title: 'str | None' = None, ref: 'str | None' = None) -> 'RegressionResult'`](https://quarcs-lab.github.io/geo4bolivia/reference/regression.html): Ordinary least squares, with several specifications side by side. - [`geo.panel_model(data, y: 'str', x, *, models=('pooled', 'between', 'fe', 're'), time_effects: 'bool' = False, cluster: 'bool' = True, title: 'str | None' = None, ref: 'str | None' = None) -> 'PanelModelResult'`](https://quarcs-lab.github.io/geo4bolivia/reference/panel_model.html): The standard panel estimators side by side, and the Hausman test of fixed against random effects. - [`geo.spatial_model(data, y: 'str', x, year=None, *, model: 'str' = 'durbin', method: 'str' = 'ml', w=None, fixed_effects: 'str | None' = None, impacts: 'bool' = True, n_draws: 'int' = 10000, seed: 'int | None' = 12345, title: 'str | None' = None, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/spatial_model.html): A spatial regression for one year, with direct, indirect (spillover) and total impacts. - [`geo.spatial_diagnostics(data, y: 'str', x, year=None, *, w=None, fixed_effects: 'str | None' = None, alpha: 'float' = 0.05, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/spatial_diagnostics.html): Tests for spatial dependence left in a least-squares regression, and the spatial model they point to. ## Spatial heterogeneity - [`geo.mgwr(data, y: 'str', x, year=None, *, kernel: 'str' = 'bisquare', criterion: 'str' = 'AICc', max_iter: 'int' = 200, tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/mgwr.html): Multiscale geographically weighted regression: where, and at which scale, does ``x`` relate to ``y``? ## Companion packages - [`geo.for_expdpy(data, scale: 'str | None' = None, years=None, variables=None, labels: 'dict | None' = None, groups=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/for_expdpy.html): ``(df, df_def)`` in the format expdpy expects. - [`geo.for_geometrics(data, scale: 'str | None' = None, years=None, variables=None, labels: 'dict | None' = None, groups=None, ref: 'str | None' = None)`](https://quarcs-lab.github.io/geo4bolivia/reference/for_geometrics.html): ``(gdf, df, df_dict)`` in the format geometrics expects. ==================================================================================================== # Analyze (https://quarcs-lab.github.io/geo4bolivia/analyze.html) ==================================================================================================== --- title: "Analyze" description: "A tour of the analysis functions on one question: did sustainable development become more equal across Bolivia's municipalities, and what role does space play?" image: images/thumbs/case-study-sdg.webp image-alt: "An Altiplano town at dawn with a school with a solar roof, a water tower and street lamps, quinoa fields in front" sidebar: false --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` This page takes one question through every family of analysis functions: **did sustainable development become more equal across Bolivia's municipalities between about 2012 and about 2024, and what role does space play?** Each step is one or two calls, read in a few sentences, with a link to the article that explains the method in full. The page is a tour, not a study: the [case study](articles/case-study-sdg.qmd) does the same work at full length. Every function returns a **result object**. Displayed, it shows its main figure and a reading of the numbers; `.fig`, `.df`, `.gt` and `.interpret()` give the parts one by one ([Get started](get-started.qmd) explains them). ```{python} #| eval: false !pip install -q "geo4bolivia[mgwr]" ``` The `[mgwr]` extra is needed only for the last step. ## The data The dataset `sdg` holds the **Sustainable Development Index** (`index_sdg`, called IDS, from 0 to 100) of the SDG Atlas 2026 for the 343 municipalities, in two periods stored as the years 2012 and 2024, with the census `population` and the `urbanization` rate. ```{python} import geo4bolivia as geo ids = geo.load("sdg", groups=["department"]) cols = ["variable", "year", "count", "mean", "std", "min", "max"] geo.describe(ids, variables=["index_sdg", "urbanization"])[cols] ``` The index rose by about ten points on average, from 50.1 to 60.1. `urbanization` exists only for 2024, so the models further down describe that year alone. ::: {.callout-note} ## Two periods, not two years "2012" and "2024" are nominal: each of the 81 indicators behind the index was measured in its own years, near those two (`geo.reference_years("sdg03")` lists them for one goal). Rates "per year" below divide by the twelve years between the nominal dates and are approximate. ::: ## 1. Spatial dependence **Question: do municipalities with similar values lie next to each other?** Global Moran's I measures it for the whole country: it is close to zero when values are placed at random, and positive when neighbours are alike. `year="all"` computes it in every year. ```{python} geo.moran(ids, "index_sdg", year="all").df ``` Moran's I is 0.43 in 2012 and 0.47 in 2024. In both periods none of the 999 random reshufflings of the values over the map gave a value as high (`p_sim` = 0.001, the smallest the test can return), so the index is clearly clustered, and no less in 2024 than in 2012. Local Moran statistics (LISA) show *where*. Each municipality is compared with its neighbours and, when the pattern is unlikely under random placement, assigned to a type: High-High (a high value among high neighbours), Low-Low, or one of the two kinds of outlier. ```{python} #| classes: g4-wide lisa = geo.lisa(ids, "index_sdg", year=2024) lisa ``` Of the 343 municipalities, 60 are High-High and 45 Low-Low; 11 are outliers and 227 are not significant. The next table counts them by department. ```{python} where = lisa.df.merge(ids.loc[ids["year"] == 2024, ["geo_id", "department"]]) where.groupby(["department", "cluster"]).size().unstack(fill_value=0) ``` The High-High municipalities are mostly in Santa Cruz (17), Cochabamba (14), La Paz (12) and Tarija (10, of its 11 municipalities); the Low-Low ones mostly in Potosí (13), La Paz (13) and Oruro (8). The map locates clusters; it does not say why they are there, and "high" and "low" are relative to the national mean of that year. → [Maps and spatial dependence](articles/spatial-dependence.qmd): class schemes, spatial weights, the permutation test, and how much the result depends on the definition of "neighbour". ## 2. Convergence **Question: did the municipalities that started lower rise faster?** β-convergence regresses each municipality's average yearly growth of the index on the logarithm of its initial level. A negative slope means that lower starting points go with faster growth. ```{python} beta = geo.beta_convergence(ids, "index_sdg") beta ``` The slope is −0.029 (standard error 0.002), and the initial level accounts for 44% of the differences in growth (R² = 0.44). At that pace half of a gap between two municipalities would close in about 20 years. Two cautions: an index with a ceiling at 100 leaves more room to rise at the bottom, and a negative slope does not by itself mean that the spread of the index fell (step 4 checks that directly). The other two convergence functions need more years than the Atlas has: - `geo.sigma_convergence` estimates the *trend* of the dispersion over time, so it needs at least three years. - `geo.convergence_clubs` follows each unit's path over time to sort units into groups that converge among themselves, so it needs at least six. Two datasets have long yearly series: `pop` (gridded population, 2001–2020) and `ntl` (night-time lights per capita, 2012–2020), both for 333 municipalities. Called on a dataset that is too short, either function says so and names these two. → [Convergence: β, σ and clubs](articles/convergence.qmd): all three methods, the spatial versions of β-convergence, and the long panels. ## 3. Transitions and mobility **Question: did the clusters of step 1 persist, and how much did municipalities change position?** `geo.lisa_transitions` runs the LISA analysis in both periods, on the same units and neighbours, and counts the moves between types. ```{python} moves = geo.lisa_transitions(ids, "index_sdg") moves ``` Rows are the type in 2012 and columns the type in 2024. 275 of the 343 municipalities (80%) have the same type in both periods. Of the 46 High-High municipalities of 2012, 39 are still High-High; of the 44 Low-Low, 30 are still Low-Low and 13 are no longer significant. No municipality moved between High-High and Low-Low. Moves to and from "Not significant" can come from p-values near the threshold, so read them with care. `geo.rank_mobility` ranks the municipalities in each period (rank 1 is the highest index) and compares the two rankings. ```{python} ranks = geo.rank_mobility(ids, "index_sdg") ranks ``` Each point is a municipality; on the diagonal it kept its rank. The Spearman correlation of the two rankings is 0.84 (1 would be an unchanged order), a municipality moved 40 positions on average (median 29), and 56% of them stayed in the same fifth of the ranking. The top fifth is the most stable (77% stayed). A rank is relative: a municipality can lose positions while its index rises, when others rose more. → [Transitions and mobility](articles/transitions.qmd): Markov and spatial Markov transition matrices, which need the `[dynamics]` extra, and these two functions in detail. ## 4. Inequality **Question: how unequal are the municipalities, and is the inequality between departments or inside them?** `geo.inequality` computes the Gini and Theil indices in each year (both are 0 when every unit has the same value) and draws the Lorenz curve. ```{python} ineq = geo.inequality(ids, "index_sdg") ineq.gt ``` Both measures fell: the Gini index from 0.065 to 0.049 and the Theil index from 0.0066 to 0.0038. With two periods the two values can be compared, but a trend cannot be tested. The values are small because the index is bounded and most municipalities lie within a few points of each other; they are not comparable with the Gini of income or population. Theil's index can be split exactly into inequality **between** groups (differences among department averages) and **within** them. ```{python} theil = geo.theil_decomposition(ids, "index_sdg", by="department") theil ``` Differences between the nine departments account for 20% of the inequality in 2012 and 15% in 2024. Most of the inequality between municipalities is found inside departments, and the part between departments shrank faster than the part within. → [Regional inequality](articles/inequality.qmd): what each measure is sensitive to, the Lorenz curve, trends on long panels, the spatial Gini, and what an inequality index of a bounded indicator can and cannot say. ## 5. Models **Question: what goes with a higher index, and is the relationship the same once neighbours are taken into account?** `geo.regression` fits ordinary least squares, with several specifications side by side. Here the index of 2024 is regressed on the urbanization rate, first alone and then with population and department fixed effects (a separate intercept for each department). ```{python} reg = geo.regression(ids, "index_sdg", [["urbanization"], ["urbanization", "population"]], fixed_effects=[None, "department"]) reg.gt ``` A municipality whose urbanization rate is ten percentage points higher has an index about 0.9 points higher (coefficient 0.094). The coefficient barely moves (0.087) when municipalities are compared only with others of the same department and population is held constant. Urbanization alone accounts for 31% of the variation of the index. Least squares assumes that the errors of neighbouring municipalities are unrelated. `geo.spatial_diagnostics` tests that on the residuals and points to the spatial model that fits the pattern it finds. ```{python} diag = geo.spatial_diagnostics(ids, "index_sdg", "urbanization") diag ``` The residuals are as clustered as the index itself (Moran's I = 0.47), so urbanization does not account for the spatial pattern of step 1. Both robust tests reject, and the one for a spatial error model is the larger (23.1 against 4.5), which is the model the function recommends. ```{python} sem = geo.spatial_model(ids, "index_sdg", "urbanization", model="error") sem ``` In the spatial error model the coefficient of urbanization is 0.081, a little lower than the least-squares one, and the spatial parameter λ is 0.67: what the model leaves unexplained is strongly correlated between neighbours. The association with urbanization holds; the neighbourhood pattern has another source, which this model does not identify. → [Regression, panel and spatial models](articles/models.qmd): fixed effects and clustered errors, panel estimators, and the spatial lag and Durbin models with their direct and spillover estimates. ## 6. Spatial heterogeneity **Question: is the association with urbanization the same everywhere?** The models of step 5 estimate one coefficient for the country. Multiscale geographically weighted regression (MGWR) fits a regression at every municipality, giving more weight to the nearest ones, so the coefficient becomes a surface that can be mapped. It also chooses how many neighbours to use (the *bandwidth*): a small bandwidth means an association that changes over short distances. The search takes about half a minute. ```{python} mg = geo.mgwr(ids, "index_sdg", "urbanization") mg.figs["urbanization"] ``` ```{python} local = mg.df.merge(ids.loc[ids["year"] == 2024, ["geo_id", "department"]]) print(mg.bw, "| R2 =", round(mg.r2, 3)) local.groupby("department")["urbanization_coef"].mean().round(2).sort_values() ``` MGWR standardizes the variables, so the coefficients are in standard deviations. The local coefficient of urbanization is positive everywhere, from 0.21 to 1.01, and significant in 326 of the 343 municipalities. It is estimated from the 90 nearest municipalities at each place, about a quarter of the country: a regional rather than a national pattern. On average it is highest in Potosí, Cochabamba and Chuquisaca (0.73 to 0.79) and lowest in Pando, Beni and Santa Cruz (0.32 to 0.38). → [Spatial heterogeneity: MGWR](articles/heterogeneity.qmd): bandwidths, the local tests, several regressors, and how to read the coefficient maps. ## What the tour found All of these are associations in the data; none shows a cause. - **More equal, by the measures used here.** Between about 2012 and about 2024 the index rose in 342 of the 343 municipalities, municipalities that started lower rose faster (β = −0.029), and the Gini and Theil indices of the index fell. - **Clustered, and no less so.** Moran's I is 0.43 in 2012 and 0.47 in 2024. Four of five municipalities have the same LISA type in both periods, and none moved between a High-High and a Low-Low cluster. - **A reshuffled middle, a stable top.** The ranking changed (rank correlation 0.84), mostly in its middle; 77% of the top fifth stayed there. - **Mostly within departments.** Differences between department averages are 15% of the inequality in 2024, down from 20%. - **Urbanization goes with a higher index, everywhere but not equally.** The coefficient is positive in the national models and in every local one, and larger in Potosí, Cochabamba and Chuquisaca than in Santa Cruz, Beni and Pando. It leaves the spatial clustering of the index unaccounted for. So the gaps between municipalities narrowed while the geography of high and low values stayed largely in place. What keeps it in place is a question these descriptive tools raise and cannot answer. ## Where next - **[Case study: the Sustainable Development Index](articles/case-study-sdg.qmd)**: the same question, at full length. - **[Articles](articles/index.qmd)**: one article per method. - **[Explore](explore.qmd)**: the first look at a variable, before any of this. - **[Function reference](reference/index.qmd)**: every function, with an example you can paste into Colab. ==================================================================================================== # Changelog (https://quarcs-lab.github.io/geo4bolivia/changelog.html) ==================================================================================================== --- title: "Changelog" description: "What changed in each release of geo4bolivia." sidebar: false --- ## 0.7.0 (2026-11-20) **`lisa` shows the Moran scatterplot and the cluster map side by side.** `.fig` is now one interactive figure with both panels and one legend: a click on a type (High-High, Low-Low, the outliers, not significant) shows or hides it on the scatterplot and on the map at once. `panels="map"` or `panels="scatter"` gives one panel alone, and the result keeps each one as `.fig_map` (new) and `.fig_scatter`. Nothing else changes: the same clusters, counts and reading. **Documentation** - The site opens in a dark theme matching the observatory's website (a toggle switches to light), with dark figures and tables. Every code box has Copy at the top and, for Python, Open in Colab at the bottom, which opens the page's notebook; each function of the reference has its own example notebook. - A night-sky background (faint stars over a blue-to-teal gradient, fixed while the page scrolls), shared with the observatory's website. ## 0.6.0 (2026-11-20) geo4bolivia becomes a self-contained package for spatial data science, in its own repository, with its own documentation site. **New: the analyses are built in.** Every function below takes a dataset id or a panel with `geo_id` and `year`, finds the boundaries, the spatial weights and the labels itself, and returns a result object with `.df`, `.fig` (Plotly), `.gt` (a table, where one helps) and `.interpret()`, a reading of the numbers in words. | Family | Functions | | --- | --- | | Explore | `correlation`, `extremes`, `distribution`, `scatter`, `trend` | | Maps | `map` gains class schemes (`scheme="fisher_jenks"`, `"quantiles"`, `"equal_interval"`, or `bins=`), several years side by side in common classes, and `change=True` | | Spatial dependence | `weights`, `connectivity`, `moran`, `lisa` | | Convergence | `beta_convergence` (ordinary and spatial models), `sigma_convergence`, `convergence_clubs` | | Transitions and mobility | `transitions`, `spatial_transitions`, `lisa_transitions`, `rank_mobility` | | Inequality | `inequality` (Gini, Theil, coefficient of variation, Lorenz curve), `theil_decomposition` | | Models | `regression`, `panel_model` (with the Hausman test), `spatial_model`, `spatial_diagnostics` | | Spatial heterogeneity | `mgwr` (multiscale geographically weighted regression) | The analysis engine is ported from the sister package [geometrics](https://quarcs-lab.github.io/geometrics/), and the exploration functions and panel models follow [expdpy](https://cmg777.github.io/expdpy/). Both remain the place for specialised methods; `for_geometrics` and `for_expdpy` still hand panels over to them. **Changed** - **Two repositories.** The package now lives in [quarcs-lab/geo4bolivia](https://github.com/quarcs-lab/geo4bolivia). The data, their builders, the notebooks and the observatory website moved to [quarcs-lab/gdo4bolivia](https://github.com/quarcs-lab/gdo4bolivia), which the package reads. - **Versions 0.5 and earlier no longer find the data**, because they look for them at the package's old address. Upgrade with `pip install -U geo4bolivia`. If you cannot upgrade, set the environment variable `GEO4BOLIVIA_BASE_URL` to `https://raw.githubusercontent.com/quarcs-lab/gdo4bolivia/{ref}/`. - **Dependencies.** The default install now includes the libraries of the core analyses: libpysal, esda, spreg, inequality, mapclassify, statsmodels, scipy, plotly and great-tables. Heavier ones are extras: `geo4bolivia[dynamics]` (giddy, for `transitions` and `spatial_transitions`), `[mgwr]` (for `mgwr`), `[panel]` (linearmodels, for `panel_model`), `[raster]` (rasterio, for `zonal`) and `[all]`. `[plot]` is kept as an empty alias, since plotly is now always installed. - **`geo4bolivia build`** runs the builders from the `builders/` folder of a checkout of the data repository; the package keeps only the generic build helpers. ## 0.5.0 (2026-11-17) Functions to understand the data: `search`, `info`, `reference_years`, `cite`, `labels`, `describe`, `add_names`, `change` and `map`; `load` with names and groups; aggregation to several scales with coverage; census-population-weighted aggregation of the SDG Atlas indicators. ## 0.4.0 (2026-11-10) The SDG Atlas 2026 (two periods, 15 goals, 81 indicators) on SDSN Bolivia's map of 343 municipalities and indigenous autonomies. ## 0.3.0 and earlier The first releases: the scale registry and geo IDs, loading, linking, aggregation, zonal statistics, and the bridges to geometrics and expdpy. ==================================================================================================== # Explore (https://quarcs-lab.github.io/geo4bolivia/explore.html) ==================================================================================================== --- title: "Explore" description: "A first pass over one indicator, the Sustainable Development Index: summaries, distributions, extremes, correlations, trends and maps, before any model." image: images/thumbs/exploring.webp image-alt: "Rolls of Andean aguayo textiles of different heights standing side by side like the bars of a chart" sidebar: false --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` Before any model, look at the data. This page takes one indicator, the **Sustainable Development Index** (IDS, from 0 to 100) of Bolivia's 343 municipalities, through every exploration function of the package, in the order you would use them on a variable you have never seen: summarise it, look at its distribution, find its extremes, see what it goes with, follow it over time, and put it on a map. Each function returns a **result object**. Displayed, it shows its figure and a reading of the numbers; `.fig`, `.df`, `.gt` and `.interpret()` give the parts one by one ([Get started](get-started.qmd) explains them). ```{python} #| eval: false !pip install -q geo4bolivia ``` ## The data The dataset `sdg` holds the index (`index_sdg`), its national ranking, the census `population` and the `urbanization` rate, in two periods stored as the years 2012 and 2024. `groups=["department"]` adds the name of each municipality's department, which several steps below use. ```{python} import geo4bolivia as geo sdg = geo.load("sdg", groups=["department"]) sdg.head(4) ``` ## 1. Summarise `geo.describe()` gives the usual summary statistics of each variable in each year, with the variable's label and unit taken from the metadata. ```{python} cols = ["variable", "unit", "year", "count", "mean", "std", "min", "median", "max"] geo.describe(sdg, variables=["index_sdg", "population", "urbanization"])[cols] ``` Three things to take from this table before going on: - The index rose: its mean went from 50.1 to 60.1, and its range from 37.2–70.5 to 47.3–77.3. - `urbanization` has a `count` of 0 in 2012: it exists only for 2024. A summary table is the fastest way to find gaps like this one. - `population` is very **skewed** (a few very large values): its mean in 2024 (33,134) is more than two and a half times its median (12,296), because a handful of cities are far larger than the typical municipality. ## 2. The distribution A mean hides the shape. `geo.distribution()` draws a histogram, and with two periods it overlays them on common bins. ```{python} geo.distribution(sdg, "index_sdg") ``` The whole distribution moved to the right by about ten points, and its shape hardly changed: the standard deviation went from 5.82 to 5.26 and the interquartile range (the width of the middle half of the municipalities) from 7.4 to 7.3. The **skewness** (zero for a symmetric distribution, positive when the long tail is on the high side) is small in both periods, 0.51 and 0.31: a short tail of high-scoring municipalities. `by=` compares groups instead of years. A box plot (`kind="box"`) shows each department's median, its middle half and its unusual municipalities: ```{python} geo.distribution(sdg, "index_sdg", by="department", kind="box") ``` In 2024 the median municipality of Tarija scores 66.8 and that of Potosí 57.0. The departments overlap a great deal, though. The standard deviation within Potosí (6.09) or Cochabamba (6.00) is larger than most of the gaps between department medians, so the department a municipality lies in says only a little about its index. With `by`, the function uses one year, the last with data; `year=2012` gives the other. ## 3. The extremes Which municipalities sit at the two ends? `geo.extremes()` ranks one year (the last, by default) and returns the `n` highest and lowest. ```{python} geo.extremes(sdg, "index_sdg", n=5) ``` The five highest are large cities: Nuestra Señora de La Paz (77.3), Tarija, Sucre, Cochabamba and El Alto. The five lowest have no urban population: three are in Potosí, and the other two are indigenous territories created since 2015 (TIOC-Raqaypampa, 47.3, and TIOC-Territorio Indígena Multiétnico, 49.7). Thirty points separate the two ends. The table behind the figure is `.df`, and `.gt` formats it. ## 4. What goes with it `geo.correlation()` computes the correlation between every pair of variables in one year. It uses only the municipalities with data for *all* the variables (**complete cases**), so that every coefficient refers to the same units. ```{python} geo.correlation(sdg, ["index_sdg", "urbanization", "population"]) ``` The index is moderately correlated with urbanization (r = 0.56) and more weakly with population (r = 0.37). The year is 2024 because that is the last year with data, and the only one in which `urbanization` exists. Pearson's coefficient measures *linear* association, and population is far from linear: `method="spearman"` uses ranks instead, and `.gt` shows both coefficients in one table. ## 5. One relationship, closely A coefficient summarises a cloud of points; `geo.scatter()` shows the cloud. Here each point is a municipality, coloured by department and sized by population. Hover over a point to see its name. ```{python} geo.scatter(sdg, "urbanization", "index_sdg", color="department", size="population") ``` The least-squares line has a slope of 0.094: ten more percentage points of urban population go with an index about 0.9 points higher, on average. The line accounts for 31% of the variation of the index (R² = 0.31), so most of it is left unexplained. The figure also shows what the coefficient cannot: 158 of the 343 municipalities have no urban population at all and pile up at zero, with indices from 47.3 to 68.7. This is an association between two variables in one year, and many other things differ between urban and rural municipalities. For a skewed variable such as population, `log_x=True` puts the axis in logarithms; `fit="lowess"` replaces the straight line with a local smooth. ## 6. Over time `geo.trend()` follows a variable over time: the mean across units with a band of one standard error, and each unit's own path as a thin grey line. `by=` draws one line per group. ```{python} geo.trend(sdg, "index_sdg", by="department") ``` The mean rose from 50.1 to 60.1, and 342 of the 343 municipalities ended higher than they started. Every department's mean rose: Pando's the most (+11.5) and Santa Cruz's the least (+7.8), and the gap between the highest and the lowest department mean narrowed from 11.9 to 10.5 points. With two periods a trend is only two points joined by a line. The same function is more telling on datasets with many years, such as night-time lights (`geo.trend("ntl", "ln_NTLpc")`, 2012 to 2020). ## 7. On the map None of the steps above used the location of the municipalities. `geo.map()` does. By default it draws the last year on a continuous colour scale: ```{python} geo.map(sdg, "index_sdg") ``` A continuous scale shows every difference, and the eye is poor at comparing shades. A **classification scheme** groups the values into a few classes, one colour each, with the limits in the legend. `"fisher_jenks"` (natural breaks) puts similar values in the same class; `"quantiles"` puts the same number of units in each; `"equal_interval"` makes classes of the same width. ```{python} geo.map(sdg, "index_sdg", scheme="fisher_jenks", k=5) ``` The top class (above 68) holds 24 municipalities, 8 of them in the department of Cochabamba; it includes the largest cities (Santa Cruz de la Sierra, El Alto, La Paz, Cochabamba). The bottom class (below 54.9) holds 58. Click a class in the legend to hide or show it. With a list of years, the maps are drawn side by side **in the same classes**, computed over both years, so a colour means the same on both maps: ```{python} geo.map(sdg, "index_sdg", year=[2012, 2024], scheme="fisher_jenks") ``` In 2012, 108 municipalities were in the lowest class (below 46.9) and 4 in the highest (above 64.3). In 2024 the lowest class is empty and the highest has 72. To see *where* the index rose most, map the change itself. `change=True` computes the last year minus the first for each municipality (as `geo.change()` does) and uses a diverging scale centred at zero; with a scheme, the classes are split at zero. ```{python} geo.map(sdg, "index_sdg", change=True, scheme="fisher_jenks") ``` The only municipality below zero is San José de Chiquitos (Santa Cruz), whose index fell by 2.7 points. The largest gains, above 13.7 points, are in 29 municipalities, most of them in Cochabamba (8), La Paz (6) and Oruro (5). Municipalities that started lower tended to gain more: the correlation between the 2012 level and the change is −0.43. ## What the exploration leaves open The maps suggest that high and low values are not scattered at random, and the last number suggests that the municipalities that started lower caught up. Neither is established by looking. Is the clustering stronger than chance would produce? Did the gaps between municipalities really narrow? What remains of the link with urbanization once neighbours are taken into account? ## Where next - **[Analyze](analyze.qmd)** answers those questions with the same panel: spatial dependence, convergence, transitions, inequality and regression. - **[Exploring the data](articles/exploring.qmd)** uses the functions of this page to answer questions about multidimensional poverty, and lists the pitfalls of these data. - **[Function reference](reference/index.qmd)**: every option of `describe`, `distribution`, `extremes`, `correlation`, `scatter`, `trend` and `map`. ==================================================================================================== # Get started (https://quarcs-lab.github.io/geo4bolivia/get-started.html) ==================================================================================================== --- title: "Get started" description: "Install geo4bolivia, find a variable, load it, map it and run a first spatial analysis, in ten minutes." image: images/hero.webp image-alt: "The Salar de Uyuni at blue hour, its salt polygons glowing rose and turquoise, with town lights and the Cordillera Real behind" sidebar: false --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` **geo4bolivia** gives you the data of the [GeoDevelopment Observatory of Bolivia](https://quarcs-lab.github.io/gdo4bolivia/) and the standard analyses of spatial data science in one package. This page takes one indicator, the Sustainable Development Index of Bolivia's 343 municipalities, from *finding it* to a *first spatial result*. ## Install In Google Colab or any Python 3.11+ environment: ```{python} #| eval: false !pip install -q geo4bolivia ``` ```{python} import geo4bolivia as geo ``` The default install covers loading, maps and the core analyses. A few methods need an extra library, which you add only when you use them: | Extra | Adds | For | | --- | --- | --- | | `geo4bolivia[dynamics]` | giddy | `transitions`, `spatial_transitions` | | `geo4bolivia[mgwr]` | mgwr | `mgwr` | | `geo4bolivia[panel]` | linearmodels | `panel_model` | | `geo4bolivia[raster]` | rasterio | `zonal` | | `geo4bolivia[all]` | all of them | everything | ## 1. Find the data `geo.datasets()` is the catalog: one row per dataset, with its years and the number of units that have data. ```{python} geo.datasets()[["dataset", "title", "years", "units", "variables"]].head(6) ``` You do not need to know where a variable lives. `geo.search()` looks through the names, labels and descriptions of every variable of every dataset: ```{python} geo.search("sustainable development index")[["dataset", "variable", "label", "years", "units"]] ``` `geo.info()` documents a dataset: what it is, its sources and their licenses, and how to cite it. ```{python} geo.info("sdg") ``` ::: {.callout-note} ## Two periods, stored as 2012 and 2024 The SDG Atlas 2026 describes two periods, "around 2012" and "around 2024". Each indicator's actual years are in the metadata: `geo.reference_years("sdg03")` lists them, and `geo.cite("sdg")` gives the citation and the credits its sources require. ::: ## 2. Load it `geo.load()` returns a **panel**: one row per unit (`geo_id`) and `year`. `groups=` adds the name of the department each municipality lies in. ```{python} ids = geo.load("sdg", variables="index_sdg", groups=["department"]) ids.head() ``` Every unit has one **geo ID** that nests by prefix: `BO-04` is the department of La Paz, `BO-04-17` its province Murillo and `BO-04-17-04` the municipality of Nuestra Señora de La Paz. The [data model article](articles/data-model.qmd) explains scales and IDs. ::: {.callout-tip collapse="true"} ## Without the package: plain URLs Every dataset is also a CSV and a Parquet file you can read with pandas alone: ```python import pandas as pd BASE = "https://raw.githubusercontent.com/quarcs-lab/gdo4bolivia/main/" ids = pd.read_csv(BASE + "data/municipality/sdg.csv") units = pd.read_csv(BASE + "geo/units.csv") # names, parents and source codes ``` ::: ## 3. Look at it `geo.describe()` summarises each variable in each year, with its label and unit: ```{python} geo.describe(ids) ``` `geo.map()` draws any variable. With a classification scheme and a list of years, it puts the two periods side by side in the same classes, so the colours are comparable: ```{python} geo.map(ids, "index_sdg", year=[2012, 2024], scheme="fisher_jenks") ``` `geo.change()` gives one row per municipality with its value, change and rank in both years: ```{python} chg = geo.change(ids, "index_sdg") chg[["name", "department", "index_sdg_2012", "index_sdg_2024", "index_sdg_change"]].nlargest(5, "index_sdg_change") ``` ## 4. Analyse it Every analysis is one call. It takes the panel (or just the dataset id), finds the boundaries and builds the spatial weights itself, and returns a **result object**. Is sustainable development clustered in space? Global Moran's I answers for the whole country: ```{python} res = geo.moran(ids, "index_sdg", year=2024) res ``` Displaying a result shows its main figure and a reading of the numbers. The parts are there to use one by one: | Attribute | What it holds | | --- | --- | | `res.fig` | the main figure (Plotly, interactive); some results have more, e.g. `fig_map` | | `res.df` | the table behind it, one row per unit or year, with `geo_id` and `name` | | `res.gt` | a formatted table, where one helps (regressions, summaries) | | `res.moran_i`, `res.p_sim`, ... | the numbers, by name | | `res.interpret()` | the reading in words; it describes associations, never causes | | `res.notes` | what the function decided for you (units left out, weights used) | ```{python} print(round(res.moran_i, 3), res.p_sim, "|", res.w_spec) ``` *Where* are the clusters? Local Moran statistics (LISA) show them on the Moran scatterplot and on the map, side by side: ```{python} #| classes: g4-wide geo.lisa(ids, "index_sdg", year=2024).fig ``` And did the municipalities that started lower catch up? β-convergence compares growth with the initial level: ```{python} conv = geo.beta_convergence(ids, "index_sdg") conv ``` ## 5. Your own choices, your own data The defaults are a starting point. Build other weights with `geo.weights()` and pass them as `w=`: ```{python} w6 = geo.weights("municipality", method="knn", k=6) # the six nearest municipalities round(geo.moran(ids, "index_sdg", w=w6).moran_i, 3) ``` The functions are not tied to the observatory's data. Any table with a `geo_id` column works, and `gdf=` supplies boundaries for units the package does not know. To bring a table of your own onto the observatory's units, see [Building your own indicators](articles/own-indicators.qmd). ## Where next - **[Explore](explore.qmd)**: distributions, correlations, trends and maps, before any model. - **[Analyze](analyze.qmd)**: spatial dependence, convergence, transitions, inequality, regression and MGWR. - **[Articles](articles/index.qmd)**: one article per method, and a full case study. - **[Function reference](reference/index.qmd)**: every function, with an example you can paste into Colab. ==================================================================================================== # For AI assistants and LLMs (https://quarcs-lab.github.io/geo4bolivia/use-with-llms.html) ==================================================================================================== --- title: "For AI assistants and LLMs" description: "How an AI assistant should install and call geo4bolivia, and the files that describe the package to it." sidebar: false --- AI assistants write better code for a package when they can read what it offers. This page is for people who work with an assistant (Claude, ChatGPT, Gemini, Copilot) and for the assistants themselves. ## Give the assistant these files | File | What it holds | Use it when | | --- | --- | --- | | [`llms.txt`](https://quarcs-lab.github.io/geo4bolivia/llms.txt) | A short index: what the package is, its pages, and every function with its signature and one-line description | You want the assistant to know what exists (small enough for any model) | | [`llms-full.txt`](https://quarcs-lab.github.io/geo4bolivia/llms-full.txt) | Every page of this site, every signature and docstring, and a tested example per function, in one file | You want the assistant to write a full analysis | Paste the address into the conversation, or attach the file. For example: > Read https://quarcs-lab.github.io/geo4bolivia/llms-full.txt. Then, with geo4bolivia in Google Colab, test whether multidimensional poverty (`sdg1_2_mpi` in dataset `sdg01`) is spatially clustered in 2024 and map the clusters. ## The essentials, for an assistant **Install and import.** The code runs in Google Colab or any Python 3.11+ environment. ```python !pip install -q geo4bolivia # extras: [dynamics], [mgwr], [panel], [raster], [all] import geo4bolivia as geo ``` **Find before you load.** Do not guess dataset or variable names: look them up. ```python geo.datasets() # the catalog: dataset ids, years, units geo.search("poverty") # variables matching words, in every dataset geo.dictionary("sdg01") # every variable of a dataset: label, description, unit geo.info("sdg01") # coverage, sources, licenses, citation ``` **The data contract.** `geo.load(dataset)` returns a long panel: one row per unit and year, with columns `geo_id`, `year` and the variables. `groups=["department"]` adds the name of the department each unit lies in. Every analysis function accepts either a dataset id or such a panel. **One call per analysis.** The functions find the boundaries, build queen contiguity weights among the units with data, and take labels from the metadata. Pass `w=geo.weights(...)` or `gdf=` only to override. ```python res = geo.moran("sdg01", "sdg1_2_mpi", year=2024) # also: lisa, beta_convergence, inequality, ... res.fig # the Plotly figure res.df # the table behind it (geo_id, name, ...) res.interpret() # a reading of the numbers in words ``` **Rules that avoid the usual mistakes** - The SDG Atlas datasets (`sdg`, `sdg01` to `sdg17`) have **two periods**, stored as years 2012 and 2024, for 343 municipalities. `pop` (2001 to 2020) and `ntl` (2012 to 2020) have 333 municipalities. - Methods that need many years (`sigma_convergence`, `convergence_clubs`, `transitions`) run on `pop` and `ntl`, not on the two SDG periods; the error message says so. - `beta_convergence` and `sigma_convergence` take a variable in **levels** and log it themselves. `ln_NTLpc` is already a logarithm: use `np.exp` first. - Link tables by **code**, never by name: municipality names repeat across departments. `geo.link(table, scale="municipality", on="", key="ine_code")`. - Use the vocabulary of the package: `variable`, `year`, `first` and `last`, `by`, `y` and `x`. - Results describe **associations**. Do not turn them into causal statements. - Cite the data: `print(geo.cite("sdg01"))` gives the citation and the credits the sources require. ## For developers of agents - The function reference is generated from the docstrings, so `help(geo.moran)` and `inspect.signature(geo.moran)` return the same content as this site. - `geo.__all__` lists the public functions; everything else is internal and may change. - The package reads its data over the network from the [data repository](https://github.com/quarcs-lab/gdo4bolivia), cached for the session. Set `GEO4BOLIVIA_DATA_DIR` to a local checkout to work offline, or `GEO4BOLIVIA_REF` to a release tag to freeze the data. ==================================================================================================== # Case study: the Sustainable Development Index, ~2012 to ~2024 (https://quarcs-lab.github.io/geo4bolivia/articles/case-study-sdg.html) ==================================================================================================== --- title: "Case study: the Sustainable Development Index, ~2012 to ~2024" description: "One complete analysis with geo4bolivia: how the Sustainable Development Index of Bolivia's 343 municipalities changed between two periods, which goals moved, whether municipalities converged, and where." image: ../images/thumbs/case-study-sdg.webp image-alt: "An Altiplano town at dawn with a school with a solar roof, a water tower and street lamps, quinoa fields in front" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` This article is a short report written with the package from the first line to the last. It asks six questions about one indicator, and each is answered by one or two functions: 1. How much did the index change, and where? 2. Which goals moved most? 3. Did municipalities converge? 4. Is development clustered in space, and did the clusters move? 5. Who moved in the ranking? 6. What goes with a higher index? The other articles explain each method. This one shows how they fit together, and ends with what the analysis found and what it does not show. ```{python} #| eval: false !pip install -q geo4bolivia ``` ```{python} import pandas as pd import plotly.express as px import geo4bolivia as geo ids = geo.load("sdg", variables=["index_sdg", "urbanization"], groups=["department"]) ``` ## The data The *Atlas de los ODS en Bolivia 2026* (Andersen & Argandoña, SDSN Bolivia) measures 81 indicators of 15 Sustainable Development Goals in Bolivia's 343 municipalities and indigenous autonomies, near the 2012 census and near the 2024 census. Each indicator is rescaled between a worst and a best value, a goal index is the mean of its rescaled indicators, and the **Sustainable Development Index** (`index_sdg`, 0 to 100) is the mean of the 15 goal indices. The Atlas has no index for goals 12 and 14. ```{python} print(geo.cite("sdg")) geo.reference_years("sdg") ``` The two periods are stored at the **nominal years** 2012 and 2024. The `~` in the table says that the index mixes indicators of different actual years: census indicators refer to 2012 and 2024, while others come from surveys or records of nearby years (`geo.reference_years("sdg03")` lists, for instance, mortality rates of 2008 and 2023 and three-year averages). "2012" and "2024" mean "around" those years throughout this article. The Atlas states no open license; the observatory uses and shares it with the permission of SDSN Bolivia. ## 1. How much did the index change, and where? ```{python} country = geo.aggregate(ids, to="country", dataset="sdg") print("weighted by population:", country["index_sdg"].round(1).tolist()) geo.describe(ids, variables="index_sdg") ``` The mean across municipalities rose from 50.1 to 60.1, about ten points, and the whole distribution moved up: the lowest value of 2024 (47.3) is close to the median of 2012 (49.6). The spread narrowed a little, with a standard deviation of 5.8 in 2012 and 5.3 in 2024. These figures count every municipality equally. Weighting each by its census population, which `geo.aggregate()` does with the rule the dataset declares, gives 57.8 and 66.8: the more populous municipalities tend to have a higher index. `geo.change()` puts both periods and their difference on one row per municipality: ```{python} chg = geo.change(ids, "index_sdg") print((chg["index_sdg_change"] > 0).sum(), "of", len(chg), "municipalities rose") ordered = chg.sort_values("index_sdg_change", ascending=False)[ ["name", "department", "index_sdg_2012", "index_sdg_2024", "index_sdg_change"]] pd.concat([ordered.head(4), ordered.tail(4)]) ``` The index rose in 342 of the 343 municipalities. The largest gains, of 17 to 20 points, are in municipalities of Oruro and Cochabamba that started near 43. San José de Chiquitos (Santa Cruz) is the only one that fell, by 2.7 points, and the two smallest gains are in Santa Cruz as well. ```{python} geo.map(ids, "index_sdg", change=True, scheme="fisher_jenks") ``` The map classes the change by natural breaks. Gains of different sizes are not scattered at random: municipalities with large gains tend to be next to each other, and so do those with small gains. Section 4 measures that. ```{python} geo.trend(ids, "index_sdg", by="department") ``` Every department rose. The averages of municipalities went up most in Pando (+11.5) and least in Santa Cruz (+7.8). Tarija has the highest average in both periods (59.6 and 67.7) and Potosí the lowest (47.7 and 57.3). With two periods, each "trend" is a line between two points. ## 2. Which goals moved most? The 15 goal indices live in 15 datasets, `sdg01` to `sdg17`. `geo.combine()` puts them on one table, and `geo.describe()` gives their means with their labels: ```{python} numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 16, 17] # no goal 12 or 14 goals = geo.combine({f"sdg{n:02d}": [f"index_sdg{n}"] for n in numbers}, scale="municipality") means = geo.describe(goals).pivot(index="label", columns="year", values="mean") means["change"] = means[2024] - means[2012] means = means.sort_values("change").round(1).reset_index() px.bar(means, x="change", y="label", hover_data=[2012, 2024], labels={"change": "Change of the mean across municipalities (points)", "label": ""}, title="Goal indices: change between ~2012 and ~2024") ``` The ten points of the overall index are an average of very different movements: - **Twelve of the 15 goal indices rose.** The largest gains are in industry, innovation and infrastructure (goal 9: 21.9 to 52.3, +30.4), quality education (goal 4, +21.5), decent work and economic growth (goal 8, +18.8) and no poverty (goal 1, +17.8). - **Climate action fell** (goal 13: 79.6 to 71.8, −7.8). - **Zero hunger and life on land stood still** (goals 2 and 15, −0.3 each). A goal index moves with its indicators and with the bounds used to rescale them, so the size of a change is comparable within a goal across municipalities more safely than across goals. `geo.dictionary("sdg13")` lists the indicators behind a goal. ## 3. Did municipalities converge? Two questions: did the municipalities that started lower grow faster (β-convergence), and did the differences between municipalities shrink? ```{python} beta = geo.beta_convergence(ids, "index_sdg") beta ``` The slope is −0.029: the lower the index around 2012, the faster it grew. Read as a speed, half of a gap between two municipalities would close in about 20 years at this pace. The starting level accounts for 44% of the differences in growth. Dispersion fell too, on every measure: ```{python} ineq = geo.inequality(ids, "index_sdg", measures=("gini", "cv")) theil = geo.theil_decomposition(ids, "index_sdg", by="department") ineq.df.merge(theil.df, on="year").round(4) ``` The Gini index went from 0.065 to 0.049 and the coefficient of variation from 0.116 to 0.088. (Both are small because an index that is an average of 15 bounded indices varies little between units; compare the two periods, not the level with other variables.) Theil's index splits into inequality **between** the nine departments and inequality **within** them. The part between departments was 20% of the total in 2012 and 15% in 2024: most of the differences are between municipalities of the same department, and more so at the end. So municipalities ended closer together, and departments did as well. The [convergence article](convergence.qmd) shows that the slope holds within departments and in spatial models, and why two periods call for caution: a unit measured unusually low in the first period looks like fast growth afterwards. ## 4. Is development clustered, and did the clusters move? Global Moran's I measures whether neighbouring municipalities have similar values (positive), or whether values are placed as if at random (near zero). ```{python} of_change = geo.moran(chg, "index_sdg_change") print("Moran's I of the change:", round(of_change.moran_i, 3), "p:", of_change.p_sim) geo.moran(ids, "index_sdg", year="all").df ``` The index is clearly clustered in both periods, and slightly more in the second: Moran's I is 0.43 and 0.47, each far from what random placement gives (pseudo p-value 0.001). The change is clustered as well, less strongly (0.22). Convergence in values did not go with a weaker spatial pattern. Local statistics (LISA) say where the clusters are, on the Moran scatterplot and on the map: ```{python} #| classes: g4-wide geo.lisa(ids, "index_sdg", year=2024).fig ``` In 2024, 60 municipalities are in **High-High** clusters (a high index among high neighbours) and 45 in **Low-Low** clusters. `geo.lisa_transitions()` compares the types of the two periods: ```{python} lt = geo.lisa_transitions(ids, "index_sdg") print(lt.n_stayed, "of", lt.n_units, "kept their type") lt.fig ``` ```{python} where = lt.df.merge(ids[["geo_id", "department"]].drop_duplicates()) pd.concat({year: pd.crosstab(where["department"], where[f"cluster_{year}"].rename("type"))[ ["High-High", "Low-Low"]] for year in (2012, 2024)}, axis=1) ``` - **The map of clusters is mostly the same**: 275 of 343 municipalities (80%) have the same type in both periods, and none went from High-High to Low-Low or back. - **High-High clusters grew** from 46 to 60 municipalities, and 39 of the 46 stayed. Around 2012 they were in Santa Cruz (23), Cochabamba (10) and Tarija (10). Around 2024 La Paz has 12 where it had 2 and Cochabamba 14, while Santa Cruz has 17. Ten of Tarija's 11 municipalities are High-High in both periods. - **Low-Low clusters kept their size**: 44 and 45 municipalities, 30 of them the same. In both periods most are in Potosí, La Paz and Oruro (30 of 44, then 34 of 45). Santa Cruz had none around 2012 and has 4 around 2024; Pando had 3 and has none. Most changes of type are to or from "Not significant", a category that depends on a p-value threshold. They are worth a look on the map (`lt.fig_map`) before being read as change. ## 5. Who moved? The index rose almost everywhere, so the ranking is the place to see who gained more than the rest. ```{python} rm = geo.rank_mobility(ids, "index_sdg") print("Spearman:", round(rm.spearman, 2), " same quintile:", round(rm.share_same_group, 2)) movers = pd.concat([rm.climbers.head(5), rm.fallers.head(5)]) movers.merge(ids[["geo_id", "department"]].drop_duplicates())[ ["name", "department", "index_sdg_2012", "index_sdg_2024", "rank_2012", "rank_2024"]] ``` The order was broadly kept: the rank correlation between the two periods is 0.84 and 56% of the municipalities are in the same fifth of the ranking. Inside that, some moves are long. Cruz de Machacamarca (Oruro) went from position 315 to 90, and the five largest climbs are in Oruro and Cochabamba, all from the bottom fifth. San José de Chiquitos went from 42 to 283, and four of the five largest falls are in Santa Cruz. Except for San José de Chiquitos, the municipalities that fell in the ranking raised their index; the others raised theirs more. ## 6. What goes with a higher index? One model, as an example of the step from description to association. The dataset has the urbanization rate of 2024 (the share of the population living in urban areas), so the model is of the level in 2024. An ordinary regression assumes that its errors are independent; with a variable this clustered, that needs checking first. `geo.spatial_diagnostics()` fits the regression and tests what is left in its residuals: ```{python} diag = geo.spatial_diagnostics(ids, "index_sdg", "urbanization", year=2024) print("recommended model:", diag.recommendation) diag.gt ``` The residuals of the ordinary regression are as clustered as the index itself (Moran's I 0.47): urbanization does not account for the spatial pattern. Both robust tests reject, the one for spatially correlated errors by far the more strongly (23.1 against 4.5), so the tests point to the **spatial error model**, which keeps the regression and lets neighbouring errors be correlated. ```{python} model = geo.spatial_model(ids, "index_sdg", "urbanization", year=2024, model="error") model.gt ``` One more percentage point of urban population goes with an index 0.08 points higher (standard error 0.007): a municipality that is 50 points more urban than another has, on average, an index about 4 points higher. The spatial parameter λ is 0.67: what the model leaves unexplained is strongly shared between neighbours. This is an association in one cross-section. Urbanization was measured in the same year as the index, several of the indicators behind the index measure access to services, which can differ between urban and rural places for many reasons, and nothing here says what would happen to a municipality if it urbanized. Urbanization is not available for 2012, so the same model cannot be estimated for the change. ## What we found All of these are associations between measured quantities. - **The index rose almost everywhere**: from 50.1 to 60.1 on average across municipalities (57.8 to 66.8 weighted by population), in 342 of 343 municipalities and in every department. - **The goals moved very differently**: twelve of 15 goal indices rose, most of all industry, innovation and infrastructure (+30.4), education (+21.5), decent work (+18.8) and poverty (+17.8); climate action fell (−7.8), and zero hunger and life on land did not move (−0.3). - **Municipalities ended closer together**: the lower the start, the faster the growth (β = −0.029, a half-life of about 20 years); the Gini index fell from 0.065 to 0.049; the share of inequality that lies between departments fell from 20% to 15%. - **The geography persisted**: Moran's I is 0.43 and 0.47; 80% of the municipalities kept their LISA type; High-High clusters grew from 46 to 60 municipalities, and Low-Low clusters stayed at 44 and 45. - **The ranking was broadly kept** (Spearman 0.84), with long moves up in Oruro and Cochabamba and down in Santa Cruz. - **More urban municipalities have a higher index** in 2024 (0.08 points per percentage point), and the part of the index that urbanization does not account for is strongly clustered (λ = 0.67). ## What this does not show - **Two periods.** Every statement about change rests on two observations per municipality. There is no trend to test, a low first value followed by a faster rise can be regression to the mean, and transitions between two dates describe one step. - **Nominal years.** "2012" and "2024" stand for indicators of different actual years. Yearly rates and the half-life use twelve years as a convention. - **An index of rescaled indicators.** Each indicator is rescaled between fixed bounds and the goals are averaged with equal weights. A different choice of bounds or weights gives a different index, and a unit near a bound has little room to move. The index says nothing about which of its 81 indicators moved in a given municipality: go to the goal datasets for that. - **Units.** Four of the 343 units were created after the 2012 census; the Atlas gives them values in both periods. Means across municipalities count a municipality of 900 people like one of 1.6 million. - **No causes.** Nothing here identifies why a municipality gained or lost. Convergence, clusters and the urbanization coefficient describe patterns in the data. ## In short - One indicator, six questions, and one or two functions for each: `describe`, `change`, `map` and `trend` for the change; `combine` for the goals; `beta_convergence`, `inequality` and `theil_decomposition` for convergence; `moran`, `lisa` and `lisa_transitions` for space; `rank_mobility` for the ranking; `spatial_diagnostics` and `spatial_model` for a model. - Every function took the same table, `ids`, and returned its figure, its table and its numbers. - The findings are descriptions of two periods. The limits are part of the result. ## Where next - **[Exploring the data](exploring.qmd)**: distributions, extremes, correlations and trends. - **[Maps and spatial dependence](spatial-dependence.qmd)**: weights, Moran's I and LISA. - **[Convergence: β, σ and clubs](convergence.qmd)**: conditional and spatial β-convergence, and methods for longer panels. - **[Transitions and mobility](transitions.qmd)**: rank mobility and LISA transitions in detail. - **[Regional inequality](inequality.qmd)**: Gini, Theil and the Lorenz curve. - **[Regression, panel and spatial models](models.qmd)**: choosing and reading a spatial model. - **[The data model](data-model.qmd)**: scales, geo IDs and the two nominal periods. ==================================================================================================== # Convergence: β, σ and clubs (https://quarcs-lab.github.io/geo4bolivia/articles/convergence.html) ==================================================================================================== --- title: "Convergence: β, σ and clubs" description: "Three questions about catching up, and the method for each: do units that start lower grow faster, is dispersion falling, and do units head for one path or for several?" image: ../images/thumbs/convergence.webp image-alt: "Meandering rose and turquoise rivers of the Bolivian lowlands joining into one wide river at dawn" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` "Convergence" is the idea that places which are behind catch up with those ahead. It is three different questions, and each has its own method: | Question | Method | Function | Years needed | | --- | --- | --- | --- | | Do units that start lower grow faster? | β-convergence | `geo.beta_convergence` | 2 | | Is the dispersion across units falling? | σ-convergence | `geo.sigma_convergence` | 3 or more | | Do units head for one common path, or for several? | Convergence clubs | `geo.convergence_clubs` | 6 or more | The answers can differ. Units that start lower can grow faster while the gaps between units stay as wide as before, and dispersion can fall across the whole country while groups of units stay apart. This article runs the three methods on the observatory's data and reads each result. ```{python} #| eval: false !pip install -q geo4bolivia ``` ```{python} import numpy as np import pandas as pd import geo4bolivia as geo ``` ## β-convergence: do units that start lower grow faster? The data are the Sustainable Development Index (`index_sdg`, 0–100) of Bolivia's 343 municipalities in two periods, stored as 2012 and 2024. ```{python} ids = geo.load("sdg", variables=["index_sdg", "population"], groups=["department"]) ``` `geo.beta_convergence()` computes each unit's average yearly growth between the two years (the difference of the logarithms, divided by the number of years) and regresses it on the logarithm of the starting level: $$\frac{\ln y_{i,\text{last}} - \ln y_{i,\text{first}}}{T} = \alpha + \beta \, \ln y_{i,\text{first}} + \varepsilon_i$$ A negative slope β is called **β-convergence**: the lower the start, the faster the growth. ```{python} beta = geo.beta_convergence(ids, "index_sdg") beta ``` The numbers behind the reading are attributes of the result: ```{python} print("slope:", round(beta.beta_total, 4), " std. error:", round(beta.se_total, 4)) print("speed:", round(beta.speed, 4), " half-life, years:", round(beta.half_life, 1)) print("R²:", round(beta.r2, 2), " units:", beta.n_obs, " years:", beta.horizon) ``` How to read them: - **The slope** is −0.0287. A municipality that started with an index 10% lower than another grew about 0.27 percentage points faster per year (0.0287 × ln 1.1). The initial level alone accounts for 44% of the differences in growth. - **The speed of convergence** turns the slope into a rate, λ = −ln(1 + βT) / T with T = 12 years: 0.035, so about 3.5% of a gap between two municipalities closes each year. - **The half-life** is the time it takes to close half of a gap at that speed, ln 2 / λ: about 20 years. Speed and half-life are a way of expressing the slope. They describe the pace seen between these two periods; they do not forecast that it will continue. `beta.df` has the regression's data, one row per municipality, and `beta.gt` the estimates as a table. ```{python} beta.df.head() ``` ### Conditional convergence The regression above compares all municipalities with each other: **absolute** convergence. **Conditional** convergence asks whether the catch-up holds among units that are alike in other respects. `controls=` adds variables measured in the first year, and `fixed_effects=` adds a dummy variable per group, here per department, so that municipalities are compared only with others of the same department. ```{python} ids["log_population"] = np.log(ids["population"]) cond = geo.beta_convergence(ids, "index_sdg", controls="log_population", fixed_effects="department") print("slope:", round(cond.beta_total, 4), " std. error:", round(cond.se_total, 4)) print("half-life, years:", round(cond.half_life, 1), " R²:", round(cond.r2, 2)) cond.fig_conditional ``` Among municipalities of the same department and of similar population in 2012, the slope is −0.0271, close to the −0.0287 found without conditions, and the half-life is about 21 years. So the catch-up is not only a matter of some departments, or of small or large municipalities, growing faster as a group: it shows up inside the departments too. The figure plots growth against the initial level after removing from both what the control and the departments account for. ### Spatial models Ordinary least squares treats each municipality as independent of the others. Neighbouring municipalities often grow together, and a model can say so. `model=` chooses one, estimated on the queen contiguity weights (units that share a border or a corner are neighbours) unless you pass your own `w=`: | `model=` | What it adds | Parameter | | --- | --- | --- | | `"sar"` | the average growth of the neighbours (spatial lag) | `rho` | | `"sem"` | errors that are correlated between neighbours (spatial error) | `lam` | | `"sdm"` | the neighbours' growth and the neighbours' initial level (spatial Durbin) | `rho` | ```{python} models = {m: geo.beta_convergence(ids, "index_sdg", model=m) for m in ["sar", "sem", "sdm"]} columns = [beta.summary.set_index("metric")["ols"]] columns += [res.summary.set_index("metric")[m] for m, res in models.items()] rows = ["direct", "indirect", "total", "rho", "lambda", "half_life", "aic"] pd.concat(columns, axis=1).loc[rows].round(4) ``` When a model includes the neighbours' growth, one unit's starting level is linked to its neighbours' growth as well as to its own, so the slope splits in two parts: - the **direct** part: how a unit's own growth goes with its own initial level; - the **indirect** part, or spillover: how a unit's growth goes with the initial levels of its neighbours; - their sum is the **total**, which `beta_total` holds and from which speed and half-life are computed. Read the table row by row: - **The direct part is negative in every model**, between −0.027 and −0.034: whatever is assumed about space, municipalities that started lower grew faster. - **Growth is spatially dependent.** The spatial parameters are positive (ρ = 0.23 in the lag model, λ = 0.47 in the error model, ρ = 0.44 in the Durbin model), and all three models have a lower AIC than OLS (AIC measures fit with a penalty for extra parameters; lower is better). - **The indirect part depends on the model.** In the lag model it has the sign of the direct part by construction, −0.008, and the total is −0.034 (half-life about 16 years). In the Durbin model it is free, and it comes out positive, +0.013: given its own start, a municipality grew faster when its neighbours started *higher*. The total is then −0.021 (half-life about 29 years). - The error model has no spillover: it gives another estimate of the same slope, −0.032, with a half-life of about 17 years. The Durbin model has the lowest AIC here, with the error model close to it. The cautious summary is a range: the half-life of a gap is between about 16 and 29 years depending on how space enters, and the catch-up itself is in every model. Displaying one of the results gives its figure and reading, for example `models["sdm"]`. ### Reasons for caution with two periods A negative β is a fact about these two periods. Several things other than catching up can produce one: - **Regression to the mean.** A municipality whose index was unusually low in the first period, for reasons that do not last, will tend to be closer to its usual level in the second, and so will seem to have grown fast. With two periods there is no way to separate this from lasting catch-up. - **Measurement error.** The initial level is on both sides of the equation: it is the regressor and it is subtracted to compute growth. Any error in the first-period value makes that unit look lower at the start *and* faster afterwards, which pushes β towards convergence. - **A bounded index.** The index runs from 0 to 100 and is built from indicators rescaled between a worst and a best value. A unit close to the best value of an indicator has little room left to rise. - **Nominal years.** The two periods are "around 2012" and "around 2024": each indicator behind the index has its own reference years (`geo.reference_years`). The 12 years used for the yearly rates are a convention. A negative β also does not mean that dispersion fell: units can swap positions without getting closer. That is a separate question. ## σ-convergence: is dispersion falling? **σ-convergence** looks at the whole distribution instead of each unit's growth: it measures the dispersion across units in every year (the standard deviation of the logarithm, and two more measures) and estimates its trend. A falling dispersion means the units are becoming more alike. A trend needs at least three years, and the SDG data have two. The function says so, and says where to go instead: ```{python} try: geo.sigma_convergence(ids, "index_sdg") except ValueError as error: print(error) ``` With two years, compare the dispersion of each directly, with `geo.inequality()`: ```{python} geo.inequality(ids, "index_sdg", measures=("gini", "cv")).df ``` Both measures are lower in the second period: the Gini index went from 0.065 to 0.049 and the coefficient of variation (the standard deviation divided by the mean) from 0.116 to 0.088. So for this index, lower starters grew faster *and* the municipalities ended closer together. Two points do not make a trend, though, and the [inequality article](inequality.qmd) goes further. ### A panel with many years Night-time lights per capita (`ntl`) have nine years, 2012 to 2020, for 333 municipalities. They are often used as a stand-in for economic activity where local accounts do not exist. ::: {.callout-important} ## Levels or logarithms? `beta_convergence` and `sigma_convergence` take the logarithm of the variable themselves, so they need it **in levels**. The dataset stores lights as a logarithm, `ln_NTLpc`, so convert it first with `np.exp`. `convergence_clubs` uses the variable as given, so it takes `ln_NTLpc` as it is. If you pass a logarithm to the first two, they warn you. ::: ```{python} ntl = geo.load("ntl") ntl = ntl.assign(ntl_pc=np.exp(ntl["ln_NTLpc"])) sigma = geo.sigma_convergence(ntl, "ntl_pc", title="σ-convergence: night-time lights per capita") sigma ``` ```{python} print(sigma.notes) sigma.df ``` The standard deviation of log lights per capita fell from 1.41 in 2012 to 1.17 in 2020, a trend of −2.8% per year, and the Gini index and the coefficient of variation fall too. Municipalities became more alike in lights per capita over these nine years. Three things to keep in mind: - **The units.** The method uses only the units observed in every year, so that the comparison is between the same units: 316 here. The note says how many were left out. Lights are missing where no pixel stayed lit after masking, so the units left out tend to be those with very little light, not a random sample. - **The path.** The fall is not steady: most of it happens between 2016 and 2017, and dispersion is slightly higher in 2019 and 2020 than in 2018. A linear trend summarises a path that the table and the figure show in full. - **The variable.** Lights per capita are a ratio of two estimates (satellite radiance and gridded population). Changes in either one move it. The slope alone can hide the shape of the path. Population (`pop`, 2001 to 2020, gridded estimates for 333 municipalities) is a clear case: ```{python} sigma_pop = geo.sigma_convergence("pop", "pop") print("trend of the log standard deviation:", round(sigma_pop.std_slope, 5)) sigma_pop.fig ``` Over the twenty years the trend is negative, −0.2% per year: municipalities became slightly more alike in population size. But the standard deviation of log population falls only until 2015 (from 1.197 to 1.148) and rises a little afterwards (1.154 in 2020). A single slope reports convergence; the figure shows that it stopped. ## Convergence clubs: one path or several? β and σ describe the country as a whole. Phillips and Sul's method asks a different question: are all units heading for **one common path**, and if not, are there **groups** of units, called clubs, that converge among themselves? It works on each unit's path relative to the average of all units in the same year. If the units converge, the dispersion of these relative paths shrinks towards zero over time, and a regression called the **log t test** checks whether it does: a t-statistic below −1.65 rejects convergence at the 5% level. When convergence is rejected for all units, the method orders the units by their final level, looks for a core group that passes the test, adds the units that pass with it, and repeats with those that are left. Units that fit nowhere are called divergent. The method follows paths over time, so it needs a real panel: at least six years with the default settings. It takes the variable as given, so pass the logarithm. ```{python} clubs = geo.convergence_clubs("ntl", "ln_NTLpc") clubs ``` ```{python} clubs.gt ``` Convergence of all 316 municipalities to one path is rejected (t = −32.2). The method finds four clubs: two large ones, of 151 and 130 municipalities, one of 30 and one of 3, and two municipalities that fit no club. For each club the table gives its own log t test: a t-statistic above −1.65 means convergence inside the club is not rejected. The mean of each club in the first and the last year shows where the clubs stand and how far apart they are: ```{python} levels = clubs.df[clubs.df["year"].isin([2012, 2020])] levels.pivot_table(index="club", columns="year", values="value").round(2) ``` Here club 1 is the brightest and club 4 the darkest; club 0 holds the two divergent units, and `value` is the trend of the logarithm (see below). Clubs 1, 2 and 3 rose, and the gap between the two large clubs widened, from 1.1 log points in 2012 to 1.3 in 2020. This is the sense in which σ-convergence and clubs can both be true: dispersion across all municipalities fell over the period, yet the test does not find a single path that all of them approach. The map shows where the members of each club are, and `clubs.membership` lists them: ```{python} clubs.fig_map ``` Read club results with these limits in mind: - **Nine years are few.** By default the test leaves out the first 30% of the years, so each log t regression rests on six points. The method was designed for longer panels. - **Paths are smoothed first.** By default each unit's path is replaced by its Hodrick–Prescott trend (`hp_filter=True`), which removes year-to-year noise. Clubs are about long-run paths, and with short panels the trend is itself uncertain. - **Membership depends on choices**: the filter, the trimming (`trim=`), the rule for merging clubs (`merge=`) and the units included. A check with the dataset's own trend variable, which is filled in for all 333 municipalities, gives a different count: ```{python} alt = geo.convergence_clubs("ntl", "ln_t400NTLpc", hp_filter=False, map=False) print(alt.n_units, "units:", alt.n_clubs, "clubs,", alt.n_divergent, "divergent") ``` Three clubs instead of four. The rejection of one common path holds in both runs; the number of clubs and who belongs to which should be treated as a description that depends on the settings. A club is a group of units whose paths move together. The method does not say why. ## In short - **β-convergence** (`geo.beta_convergence`, two years): the Sustainable Development Index grew faster where it started lower, with a slope of −0.029 and a half-life of about 20 years; the result holds within departments and in the spatial models, where the half-life ranges from about 16 to 29 years. - With two periods, regression to the mean and measurement error can produce a negative β on their own. Check dispersion as well. - **σ-convergence** (`geo.sigma_convergence`, three or more years): the dispersion of log night-time lights per capita fell by 2.8% per year between 2012 and 2020 among 316 municipalities. Look at the path as well as the slope. - **Convergence clubs** (`geo.convergence_clubs`, six or more years): one common path for lights per capita is rejected; the method finds four clubs, or three with other settings. - `beta_convergence` and `sigma_convergence` take levels; `convergence_clubs` takes the variable as given. - All of these are descriptions of how the units moved. None of them says what made them move. ## Where next - **[Transitions and mobility](transitions.qmd)**: which units moved inside the distribution, and how far. - **[Regional inequality](inequality.qmd)**: dispersion in each year, the Lorenz curve, and inequality between and within departments. - **[Maps and spatial dependence](spatial-dependence.qmd)**: the weights behind the spatial models, Moran's I and LISA. - **[Regression, panel and spatial models](models.qmd)**: direct and indirect impacts in general, and how to choose a spatial model. - **[Case study: the Sustainable Development Index](case-study-sdg.qmd)**: convergence as one step of a complete analysis. ==================================================================================================== # The data model: scales, geo IDs, datasets and metadata (https://quarcs-lab.github.io/geo4bolivia/articles/data-model.html) ==================================================================================================== --- title: "The data model: scales, geo IDs, datasets and metadata" description: "How the observatory's data are organised (spatial scales, geo IDs, panels and metadata) and how the package reads them." image: ../images/thumbs/data-model.webp image-alt: "Agricultural terraces on Isla del Sol at dusk, nested in levels edged with turquoise light, above Lake Titicaca" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` Every function of geo4bolivia rests on a small set of conventions in the [data repository](https://github.com/quarcs-lab/gdo4bolivia). Once you know them, you can predict what any function returns, join any two tables, and read the data without the package at all. Four rules hold everything together: 1. **Every spatial unit has one geo ID** (`geo_id`), at every scale, and the IDs nest by prefix. 2. **Every dataset is a long panel**: one row per unit and year, one column per variable. 3. **The metadata are the source of truth**: each variable's label, definition, unit, source and aggregation rule are written once, and everything else reads them. 4. **Nothing published is edited by hand**: the files are generated from the original inputs and checked by a validator. ```{python} #| eval: false !pip install -q geo4bolivia ``` ```{python} import geo4bolivia as geo ``` ## 1. Scales A **scale** is a set of units that covers the country: departments, provinces, municipalities, and later grid cells. The scales are listed in a registry, which `geo.scales()` returns as a table. ```{python} geo.scales() ``` | Column | Meaning | | --- | --- | | `kind` | `admin`: administrative units that nest by ID prefix. `grouping`: unions of whole units of another scale. `grid`: regular cells. `custom`: any other zones | | `parent` | The next coarser scale every unit lies in exactly. It defines the hierarchy `geo.aggregate()` climbs | | `id_pattern` | A regular expression every ID of the scale matches. The patterns do not overlap, so an ID tells its scale | | `status` | `active`: published. `planned`: the scale and its ID scheme are reserved, with no data yet | Four scales are active: 1 country, 9 departments, 113 provinces and 343 municipalities. Two helper functions answer the usual questions about the hierarchy: what a scale nests in, and what scale an ID belongs to. ```{python} print(geo.ancestors("municipality")) print(geo.scale_of("BO-04-17"), "|", geo.scale_of("BO-04-17-04")) ``` ## 2. Geo IDs A geo ID is the country code followed by one two-digit number per administrative level. Each ID contains the IDs of the units it lies in: | Scale | Pattern | Example | | --- | --- | --- | | `country` | `BO` | `BO` (Bolivia) | | `department` | `BO-NN` | `BO-04` (La Paz) | | `province` | `BO-NN-NN` | `BO-04-17` (Murillo) | | `municipality` | `BO-NN-NN-NN` | `BO-04-17-04` (Nuestra Señora de La Paz) | `geo.units()` lists the units of a scale. Each row carries the ID of every coarser unit it lies in (`province_id`, `department_id`, `country_id`), so a join across scales never needs string surgery: ```{python} units = geo.units("municipality") units[units["geo_id"] == "BO-04-17-04"].T ``` The same row shows the **source keys**: the codes other sources use for this unit. They are kept as columns and never used as the identifier. | Column | Code of | | --- | --- | | `ine_code` | Bolivia's National Statistics Institute (INE), as text with leading zeros (`020101`) | | `mun_id` | the same INE code as a number (`20101`), as many tables store it | | `gid` | GADM 4.1 (`BOL.4.17.4_2`) | | `asdf_id` | the polygon of the DS4Bolivia repository on which `pop`, `ntl` and the embeddings were computed; empty for the 10 units whose territory changed (section 4) | | `name_ds4bolivia`, `name_gadm` | older spellings of the name | ::: {.callout-warning} ## Geo IDs are not INE codes The numbers in a geo ID follow the order of GADM, a global database of administrative areas. INE orders the departments differently, so the two codes of a unit do not match: ::: ```{python} geo.units("department")[["geo_id", "name", "ine_code", "gid"]] ``` La Paz is `BO-04` here and `02` for INE; the province Murillo is `BO-04-17` here and `0201` for INE. The numbers are not always consecutive either: the provinces of La Paz run from `BO-04-02` to `BO-04-21`, because GADM skips number 1. Treat a geo ID as a label that nests, and take the official codes from `ine_code`. Why not use names? Because they repeat: ```{python} repeated = units[units.duplicated("name", keep=False)] repeated.groupby("name")["department_id"].agg(list) ``` Eight municipality names belong to two or three municipalities in different departments, and four provinces are called Cercado. To attach a table of your own to the geo IDs, match on a code; [Building your own indicators](../articles/own-indicators.qmd) shows how `geo.link()` does it. ## 3. Units, boundaries and crosswalks The geography is three kinds of table, all keyed by `geo_id`. **Units.** Without an argument, `geo.units()` returns every unit of every scale: 466 rows. ```{python} geo.units().groupby("scale").size() ``` **Boundaries.** `geo.boundaries()` returns the polygons of a scale as a GeoDataFrame (a table with a `geometry` column, from the geopandas library), in longitude and latitude (WGS 84): ```{python} departments = geo.boundaries("department") print(type(departments).__name__, "|", departments.crs.name) print(list(departments.columns)) ``` **Crosswalks.** `geo.crosswalk()` links the units of a finer scale to those of a coarser one. `weight_area` is the share of the finer unit's area that lies in the coarser unit; it is 1 for scales that nest. ```{python} geo.crosswalk("municipality", "department").head(3) ``` ### The 343-unit map The boundaries are SDSN Bolivia's municipal map of 2025: 343 municipalities and indigenous autonomies, the units of the SDG Atlas 2026. They are lightly simplified (about 100 m), and provinces, departments and the country are exact unions of the municipalities, so the scales nest without gaps or overlaps. The map states no open license and is used and shared with SDSN Bolivia's permission; credit it as *SDSN Bolivia (2025). Límites Municipales Bolivia 2025 [Archivo shapefile]. https://sdsnbolivia.org/datos-espaciales/*. Municipalities are created over time, and the IDs are built to survive that. The previous map, of 2015, had 339 municipalities. All 339 kept their geo IDs in the 2025 map, and the four units created since were numbered after the last municipality of their province (one of them, TIOC-Territorio Indígena Multiétnico, is alone in a new province, `BO-03-09`). An ID therefore never changes meaning. What did change is the territory of the six municipalities that gave land to the new units. These ten units are the ones without an `asdf_id`: ```{python} changed = units[units["asdf_id"].isna()] changed[["geo_id", "name", "ine_code", "area_km2"]] ``` ### Holes in the map The map leaves the salt flats of Uyuni and Coipasa and lakes Poopó and Uru Uru outside every municipality. They are therefore holes at every scale, and `area_km2` does not count them. A map of the municipal areas shows them as gaps in the south-west: ```{python} geo.map(units, "area_km2", scheme="quantiles", title="Area of the municipalities (km²)") ``` The areas of the 343 municipalities add up to the area of the country unit, which confirms that the scales nest exactly: ```{python} print(round(units["area_km2"].sum()), round(geo.units("country")["area_km2"].sum())) ``` ## 4. Datasets are panels A **dataset** is a set of variables from one source. `geo.datasets()` is the catalog: ```{python} geo.datasets()[["dataset", "scale", "title", "years", "units", "variables"]] ``` `geo.load()` returns a dataset as a **long panel**: one row per unit (`geo_id`) and `year`, one column per variable. Every dataset has this shape, which is what lets the same functions work on all of them. ```{python} geo.load("sdg").head(4) ``` The arguments of `load` select and decorate the panel; they do not change its shape, with two exceptions noted below. | Argument | Effect | | --- | --- | | `years=` | one year (`2024`), a list (`[2012, 2024]`) or an inclusive range as a tuple (`(2015, 2020)`) | | `variables=` | one variable or a list; `geo_id` and `year` are always kept | | `names=True` | adds the unit's `name` | | `groups=[...]` | adds the names of the coarser units each unit lies in, e.g. a `department` column | | `wide=` | **changes the shape**: one row per unit, one column per year of the variable given | | `geometry=True` | **changes the type**: a GeoDataFrame with the boundaries attached | ```{python} geo.load("sdg", variables="index_sdg", years=2024, groups=["province", "department"]).head(3) ``` ```{python} geo.load("pop", wide="pop").iloc[:3, :5] ``` ```{python} gdf = geo.load("ntl", years=2020, geometry=True) print(type(gdf).__name__, gdf.shape) ``` A year or a variable that does not exist raises an error that says what the dataset has: ```{python} try: geo.load("sdg", years=2015) except ValueError as error: print(error) ``` Two features of the catalog above need explaining. **Two periods stored as 2012 and 2024.** The SDG Atlas measures each indicator twice, near the census of 2012 and near the census of 2024, from sources of different years. Its datasets (`sdg` and one per goal, `sdg01` to `sdg17`, with none for goals 12 and 14) store the two periods at the *nominal* years 2012 and 2024, so that every variable can be compared between them. The actual years are in the metadata (next section). **333 units instead of 343.** `pop`, `ntl`, `emb` and `embpw` were computed on the polygons of the 2015 map. Their values are valid for the 333 municipalities whose territory has not changed, and they have no rows for the ten units listed in section 3. The analysis functions work on the units with data: a spatial function, for example, builds the neighbour relations among them and reports their number in the result's `w_spec`. ## 5. The metadata Each dataset comes with a metadata file, a [Frictionless Data Package](https://datapackage.org/), that describes the dataset and every one of its columns. The package reads it whenever it needs to know something about a variable. You read it through five functions. `geo.dictionary()` is the variable dictionary: ```{python} cols = ["variable", "label", "unit", "role", "aggregation", "reference_years"] geo.dictionary("sdg")[cols] ``` | Column | Meaning | | --- | --- | | `label` | a readable name, used on axes, legends and tables | | `description` | how the variable was built (not shown above: it is long) | | `unit` | the unit of measure | | `role` | `id`, `time`, `source_key` (a code kept from the source) or `variable` | | `aggregation` | how the variable moves to coarser scales | | `reference_years` | the actual years behind each stored year, when they differ | **Aggregation rules.** A population can be summed over the municipalities of a department; a rate cannot. The rule of each variable tells `geo.aggregate()` and `geo.combine()` what to do: | Rule | For | Examples | | --- | --- | --- | | `sum` | counts and totals | `population`, `pop` | | `weighted_mean:` | rates and averages, weighted by another variable | every SDG indicator and index, weighted by census `population` | | `mean`, `min`, `max` | unweighted summaries | to be used with care | | `none` | values that cannot be aggregated without more data | logarithms (`ln_NTLpc`), rankings (`rank_sdg`), densities | The SDG indicators are weighted by the census population of the same period, which the package loads from the dataset `sdg` by itself. The department values it produces approximate, but are not, the Atlas's own department figures. **Reference years.** `geo.reference_years()` gives the actual years behind each stored year. In the hunger goal, for example, the nutrition indicators come from the health surveys of 2008 and 2023, and the agricultural ones are three-year averages: ```{python} geo.reference_years("sdg02").drop_duplicates("reference") ``` **Labels.** `geo.labels()` returns the labels as a dictionary, for the `labels=` argument of Plotly Express. The package's own figures use them without being asked. ```{python} geo.labels("ntl") ``` **Documentation and licenses.** `geo.info()` gathers what a dataset is: description, coverage, actual years, the license of the published files, the sources with their own licenses, the citation and notes on its use. ```{python} print(geo.info("ntl")) ``` Two levels of license appear. The published files are under CC BY 4.0. Each *source* keeps its own terms: the SDG Atlas 2026 states no open license and is used and shared with SDSN Bolivia's permission, as `geo.info("sdg")` reports. **Citation.** `geo.cite()` returns the dataset's citation, in APA style or BibTeX, followed by the credits its sources ask for: ```{python} print(geo.cite("sdg")) ``` ## 6. Where the data are read from The package does not carry the data. It reads the files of the data repository when you ask for them: | Where you run it | What it reads | | --- | --- | | Anywhere (Google Colab, an installed package) | the files on GitHub, from the branch `main`; each file is downloaded once per session | | With `ref="..."` in any function that reads data | the same files at a git tag or commit, which freezes the data for a reproducible analysis | | With the environment variable `GEO4BOLIVIA_REF` | that tag or commit, as the default for every call | | Inside a clone of the data repository, or with `GEO4BOLIVIA_DATA_DIR` pointing to one | the local files, with no download | ```python geo.load("ntl") # the current data (branch main) geo.load("ntl", ref="v0.5.0") # the data as they were at a release tag ``` `geo.config()` shows the settings in use, among them the repository and the default branch: ```{python} config = geo.config() print(config["repository"], "|", config["default_ref"]) ``` ### Without the package Because the data are plain files with stable paths, any tool can read them. There are three kinds of file: | Path | Contents | | --- | --- | | `data//.csv` and `.parquet` | the panel of a dataset | | `geo/units.csv`, `geo/boundaries/.geojson`, `geo/crosswalks/__.csv` | the geography | | `metadata/.datapackage.json` | the metadata | The two ways of loading the same panel, side by side: ::: {.panel-tabset} ## With the package ```python !pip install -q geo4bolivia import geo4bolivia as geo ntl = geo.load("ntl", groups=["department"]) # panel + names gdf = geo.load("ntl", years=2020, geometry=True) # + boundaries ``` ## Plain URLs ```python import pandas as pd import geopandas as gpd BASE = "https://raw.githubusercontent.com/quarcs-lab/gdo4bolivia/main/" ntl = pd.read_csv(BASE + "data/municipality/ntl.csv") # geo_id, year, variables units = pd.read_csv(BASE + "geo/units.csv", dtype={"ine_code": str}) ntl = ntl.merge(units[["geo_id", "name", "department_id"]], on="geo_id") shapes = gpd.read_file(BASE + "geo/boundaries/municipality.geojson") gdf = shapes.merge(ntl[ntl["year"] == 2020], on="geo_id") ``` ::: Replace `main` in the URL with a release tag for a frozen version. What the package adds to the plain files is the metadata at hand (labels, rules, reference years, citations), the joins done for you, and the analyses. ## In short - A **geo ID** identifies each unit and nests by prefix (`BO-04-17-04` lies in `BO-04-17`, in `BO-04`, in `BO`). It follows GADM's numbering, not INE's; the INE codes are columns of `geo.units()`. - Four scales are active (country, department, province, municipality); `geo.scales()`, `geo.ancestors()` and `geo.scale_of()` describe the hierarchy. - The boundaries are SDSN Bolivia's map of 2025 with 343 units. Salt flats and lakes are holes in it. Datasets computed on the 2015 map cover 333 units. - Every dataset is a **panel** with one row per `geo_id` and `year`. The SDG datasets store two periods at the nominal years 2012 and 2024. - The **metadata** give each variable its label, unit, aggregation rule and reference years, and each dataset its sources, licenses and citation: `geo.dictionary()`, `geo.reference_years()`, `geo.labels()`, `geo.info()`, `geo.cite()`. - The data are read from GitHub by default, from a tag with `ref=`, from a local clone, or with plain URLs and no package. ## Where next - **[Exploring the data](../articles/exploring.qmd)**: these conventions at work on multidimensional poverty, with the pitfalls they explain. - **[Building your own indicators](../articles/own-indicators.qmd)**: bring a table or a raster of your own onto the geo IDs, and move data across scales. - **[Get started](../get-started.qmd)**: from finding a variable to a first spatial result. - **[Function reference](../reference/index.qmd)**: every option of the functions used here. ==================================================================================================== # Exploring the data (https://quarcs-lab.github.io/geo4bolivia/articles/exploring.html) ==================================================================================================== --- title: "Exploring the data" description: "A question-driven exploration of multidimensional poverty in Bolivia's municipalities, and the pitfalls of these data that a newcomer should know." image: ../images/thumbs/exploring.webp image-alt: "Rolls of Andean aguayo textiles of different heights standing side by side like the bars of a chart" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` The [Explore](../explore.qmd) page tours the exploration functions one by one. This article uses them the way an analysis does: it starts from questions about one indicator, **multidimensional poverty**, and picks the function that answers each. Along the way it points out six pitfalls of these data, which are collected in a table at the end. The questions: 1. What exactly is the variable? 2. Where is poverty highest and lowest? 3. How did its distribution shift between the two periods? 4. How does it differ by department? 5. What is it associated with? 6. What does the trend of night-time lights add? ```{python} #| eval: false !pip install -q geo4bolivia ``` ```{python} import textwrap import numpy as np import geo4bolivia as geo ``` ## 1. What exactly is the variable? Exploring a variable you have not read about is how wrong conclusions start. `geo.search()` finds the candidates: ```{python} geo.search("multidimensional poverty")[["dataset", "variable", "label", "years"]] ``` The first one, `sdg1_2_mpi` in the dataset `sdg01`, is the index itself. `geo.dictionary()` documents every variable of the dataset: its label, how it was built, its unit, how it aggregates to coarser scales and the years it really refers to. ```{python} d = geo.dictionary("sdg01") d[["variable", "label", "aggregation", "reference_years"]].iloc[3:] ``` ```{python} text = d.set_index("variable").loc["sdg1_2_mpi", "description"] print(textwrap.fill(text.split(" Source:")[0], 84)) ``` So the index runs from 0 to 1, **higher means poorer**, and it combines how many households are deprived (H) with how deprived they are (A). The rest of the description (cut here at its source) says that it comes from the censuses of 2012 and 2024. ::: {.callout-warning} ## Pitfall 1: the two periods are nominal years The SDG Atlas 2026 describes two periods, "around 2012" and "around 2024", and every one of its datasets stores them as the years 2012 and 2024. The actual years differ by indicator. For multidimensional poverty they are the census years. For energy poverty (`sdg1_4_eep`), in the same dataset, the first period is 2016, so its "2012 to 2024" change covers eight years, not twelve. Check `geo.reference_years()` before computing a rate of change or comparing the changes of two indicators. ::: ```{python} geo.reference_years("sdg01", "sdg1_4_eep") ``` ## 2. Where is poverty highest and lowest? ```{python} mpi = geo.load("sdg01", variables="sdg1_2_mpi", groups=["department"]) geo.extremes(mpi, "sdg1_2_mpi", year=2024, n=5) ``` In 2024 the index ranges from 0.003 in Nuestra Señora de La Paz to 0.296 in TIOC-Raqaypampa, and the median municipality has 0.053. The lowest values are in the largest cities (La Paz, El Alto, Santa Cruz de la Sierra) and two smaller municipalities, Colcapirhua and Trigal. The highest are spread over four departments: Cochabamba, Chuquisaca, Potosí and Oruro. TIOC-Raqaypampa was also the highest in 2012 (0.563; `year=2012` shows that ranking). A ranking says nothing about where these municipalities are. The map does, and with the two periods in the same classes it also shows the size of the change: ```{python} geo.map(mpi, "sdg1_2_mpi", year=[2012, 2024], scheme="fisher_jenks") ``` The lowest class (up to 0.084) held 20 municipalities in 2012 and holds 261 in 2024. The two highest classes (above 0.273) held 113 in 2012 and hold one in 2024. ## 3. How did the distribution shift? ```{python} geo.distribution(mpi, "sdg1_2_mpi") ``` The mean fell from 0.234 to 0.063 and the whole distribution moved towards zero. The dictionary gives two reference points: the Atlas counts the target as met at 0.066 and sees major challenges from 0.23 on. In 2012, 12 municipalities were at or below 0.066 and 164 at or above 0.23; in 2024 the numbers are 221 and 1. The shape changed as well. In 2012 the distribution was fairly symmetric (skewness 0.46). In 2024 it is squeezed against its floor of zero with a tail of high values (skewness 1.44). Did the gaps between municipalities narrow? It depends on how you measure them, and `geo.change()`, which gives one row per municipality, shows why: ```{python} chg = geo.change(mpi, "sdg1_2_mpi") cols = ["sdg1_2_mpi_2012", "sdg1_2_mpi_change", "sdg1_2_mpi_pct_change"] print((chg["sdg1_2_mpi_change"] < 0).sum(), "of", len(chg), "municipalities fell") chg[cols].corr().round(2) ``` Poverty fell in all 343 municipalities. In absolute terms it fell most where it was highest: the correlation between the 2012 level and the change is −0.95. In relative terms it was the other way round: the percentage fall (76% in the median municipality) was somewhat *smaller* where poverty was higher (the correlation of the level with the percentage change is +0.38, and a less negative change is a smaller fall). ::: {.callout-warning} ## Pitfall 2: skewed variables, and what logs do For a variable that is bounded at zero and skewed, absolute and relative gaps can tell opposite stories. The standard deviation of the index fell from 0.106 to 0.044: absolute gaps narrowed. The standard deviation of its **logarithm**, which measures relative gaps, rose: ::: ```{python} mpi["ln_mpi"] = np.log(mpi["sdg1_2_mpi"]) geo.distribution(mpi, "ln_mpi").df[["year", "mean", "sd", "skewness"]].round(2) ``` It went from 0.53 to 0.75. Municipalities are closer to each other in points of the index and further apart in proportion. Neither reading is wrong; say which one you use. Variables that are already logs (night-time lights, below) describe relative differences from the start. ## 4. How does it differ by department? ```{python} geo.distribution(mpi, "sdg1_2_mpi", by="department", kind="box") ``` In 2024 the median municipality is poorest in Potosí (0.078) and least poor in Tarija (0.023) and Santa Cruz (0.027). The spread *within* departments is large: Cochabamba holds both the second lowest municipality of the country (Colcapirhua, 0.005) and the highest (TIOC-Raqaypampa, 0.296). This figure compares municipalities, each counting the same. It does not give the poverty of a department's *population*. ::: {.callout-warning} ## Pitfall 3: the mean of the municipalities is not the department's value A department's value has to weight each municipality by its people. `geo.aggregate()` does that with the rule each variable declares in the metadata: for the SDG indicators, a mean weighted by census population (`weighted_mean:population`; the weights are loaded from the dataset `sdg`). The results approximate the department values; they are not the Atlas's own department figures. ::: ```{python} dep = geo.aggregate(mpi, to="department", dataset="sdg01", variables="sdg1_2_mpi") dep = geo.add_names(dep) simple = mpi.groupby(["department", "year"], as_index=False)["sdg1_2_mpi"].mean() dep = dep.merge(simple, left_on=["name", "year"], right_on=["department", "year"], suffixes=("_weighted", "_simple")) dep.loc[dep["year"] == 2024, ["name", "sdg1_2_mpi_weighted", "sdg1_2_mpi_simple", "n_units"]].round(3) ``` The two columns differ by a factor of about two in several departments. In La Paz, the simple mean of the 87 municipalities is 0.060 and the population-weighted value 0.027, because more than half of the department's people live in two municipalities with very low poverty (El Alto and Nuestra Señora de La Paz). The same holds for the country: aggregating with `to="country"` gives 0.133 in 2012 and 0.033 in 2024, against municipal means of 0.234 and 0.063. Every "mean" in the exploration functions is a mean across municipalities. ## 5. What is it associated with? Start inside the dataset. With a dataset id, `geo.correlation()` uses every variable of it: ```{python} geo.correlation("sdg01", year=2024) ``` Multidimensional poverty moves closely with unsatisfied basic needs (r = 0.82), INE's own measure of deprivation; less closely with energy poverty (0.54); and negatively with access to the three basic services (−0.69). Its correlation with the goal index `index_sdg1` (−0.82) is not a finding: the index is built from these four indicators, so it is correlated with them by construction. Other candidates live in other datasets. `geo.combine()` puts several datasets on one scale, as one panel; a dictionary picks the variables. ```{python} both = geo.combine({"sdg01": ["sdg1_2_mpi"], "sdg": ["index_sdg", "urbanization", "population"]}, scale="municipality") both = geo.add_names(both, groups=["department"]) variables = ["sdg1_2_mpi", "index_sdg", "urbanization", "population"] res = geo.correlation(both, variables) res.gt ``` The table shows Pearson coefficients above the diagonal and Spearman (rank) coefficients below. Poverty is lower where the Sustainable Development Index is higher (−0.73; again partly by construction, since the index averages the goal indices and the poverty goal is one of them), where the urbanization rate is higher (−0.46) and, weakly, where the population is larger (−0.20). ::: {.callout-warning} ## Pitfall 4: correlations use complete cases `geo.correlation()` keeps only the rows with data for *every* variable, so that all coefficients refer to the same units. One variable with gaps therefore removes rows from all the pairs. `urbanization` exists only for 2024, so pooling both periods with `year="all"` uses half of the panel, and the result says so in `.notes`: ::: ```{python} pooled = geo.correlation(both, variables, year="all") print(pooled.n, "observations |", pooled.notes[0]) ``` Population is the kind of variable for which a straight line on the raw scale is a poor summary: the largest city has about 130 times the population of the median municipality. `log_x=True` puts the horizontal axis in logarithms. ```{python} geo.scatter(both, "population", "sdg1_2_mpi", log_x=True, color="department") ``` Even in logs the association is weak (r = −0.23): the line accounts for 5% of the variation in poverty. The 17 municipalities with more than 100,000 people mostly have very low poverty (their median is 0.014), while the 55 with fewer than 5,000 range from 0.006 to 0.177. These are associations between municipal averages in one year, and they say nothing about why some municipalities are poorer. ## 6. What does the trend of night-time lights add? Poverty is observed twice, twelve years apart. Night-time lights, measured by satellite, are observed every year from 2012 to 2020; the dataset's description calls them a widely used proxy for local economic activity and electrification. The variable `ln_NTLpc` is the logarithm of radiance per person. ```{python} ntl = geo.load("ntl", groups=["department"]) res = geo.trend(ntl, "ln_NTLpc") res ``` The mean of the logarithm rose from 13.1 to 13.9. A difference of 0.81 in a logarithm is a multiplication by about 2.25 (`exp(0.81)`), and 94% of the municipalities with data in both years ended higher. Most of the rise came in the first years: the mean reached 13.9 in 2017 and has stayed near that level since. That is what an annual series adds to two census snapshots: the path between them. Two more pitfalls come with this dataset. ::: {.callout-warning} ## Pitfall 5: 333 units, not 343, and gaps within them `pop`, `ntl` and the satellite embeddings were computed on the municipal map of 2015. They are published for the 333 municipalities whose territory has not changed since, and not for the four units created since then or the six that lost territory to them. On top of that, `ln_NTLpc` is missing in the years in which no lit pixel remained in a municipality. ::: ```{python} print(res.notes[0]) units = geo.units("municipality") missing = units.loc[~units["geo_id"].isin(ntl["geo_id"]), ["geo_id"]] missing.merge(mpi[mpi["year"] == 2024]).sort_values("sdg1_2_mpi")[ ["geo_id", "name", "department", "sdg1_2_mpi"]] ``` The ten units left out are not a random sample: they include TIOC-Raqaypampa, the poorest municipality in both periods. And because the number of municipalities with data grows from 318 in 2012 to 331 in 2019, part of the trend comes from dimly lit municipalities entering the sample. The dataset has a second variable for this, `ln_t400NTLpc`, a smoothed trend that is complete for all 333 units in every year; on it, the mean rises by 1.01 instead of 0.81. How do lights relate to poverty? The two datasets share one year, 2012, and `geo.scatter()` uses it when asked: ```{python} lights = geo.combine({"sdg01": ["sdg1_2_mpi"], "ntl": ["ln_NTLpc"]}, scale="municipality") geo.scatter(lights, "ln_NTLpc", "sdg1_2_mpi", year=2012) ``` Across the 318 municipalities with both values, poverty is lower where lights per person are higher (r = −0.58): one more unit of the logarithm (2.7 times the radiance per person) goes with an index 0.043 lower, on average. Lights account for about a third of the variation in poverty (R² = 0.34), so they are an informative but rough proxy. ::: {.callout-warning} ## Pitfall 6: not every variable can be aggregated A mean of logarithms is not the logarithm of the department's lights per person, so the metadata gives `ln_NTLpc` the rule `none` and `geo.aggregate()` refuses to guess: ::: ```{python} try: geo.aggregate(ntl, to="department", dataset="ntl", variables="ln_NTLpc") except ValueError as error: print(error) ``` To compare departments, either ask for the mean of the municipalities explicitly (`rules={"ln_NTLpc": "mean"}`) and describe it as such, or rebuild the totals (lights = exp(`ln_NTLpc`) × population), sum them and take the logarithm again, as [Building your own indicators](../articles/own-indicators.qmd) does. The result's `n_units` and `coverage` columns tell how many municipalities each department value rests on. ## The pitfalls, in one table | Pitfall | Where it shows | What to do | | --- | --- | --- | | Two periods stored at nominal years | every `sdg` dataset | `geo.reference_years()` before computing rates or comparing changes | | Skewed variables | poverty near zero, population | look at the distribution; use logs or ranks for relative differences, and say which | | Municipal means are not department values | any `by="department"` summary | `geo.aggregate()` with the metadata rules; they approximate the official figures | | Complete cases | `geo.correlation()` with a variable that has gaps | read `.n` and `.notes`; correlate fewer variables or one year | | 333 versus 343 units | `pop`, `ntl`, `emb`, `embpw` | check which units are missing; they are not a random sample | | Variables without an aggregation rule | logs, trends, ranks | aggregate the underlying totals, or state an explicit rule | ## In short - Multidimensional poverty fell in all 343 municipalities between the censuses of 2012 and 2024; the municipal mean went from 0.234 to 0.063. - Absolute gaps between municipalities narrowed and relative gaps widened: the distribution is now skewed, with a tail of high values. - Departments differ, and municipalities within a department differ more. Population-weighted department values are about half the simple municipal means in several departments. - Poverty is associated with unsatisfied basic needs, lower urbanization and lower night-time lights per person. None of these associations says what causes what. - Read the metadata first: the reference years, the units covered and the aggregation rule decide what a number means. ## Where next - **[The data model](../articles/data-model.qmd)**: scales, geo IDs, the metadata and the 343-unit map. - **[Maps and spatial dependence](../articles/spatial-dependence.qmd)**: is poverty clustered in space, and where? - **[Regional inequality](../articles/inequality.qmd)**: measuring the gaps this article described by eye. - **[Convergence](../articles/convergence.qmd)**: did the municipalities that started worse off catch up? - **[Explore](../explore.qmd)**: the same functions, one by one. ==================================================================================================== # Spatial heterogeneity: MGWR (https://quarcs-lab.github.io/geo4bolivia/articles/heterogeneity.html) ==================================================================================================== --- title: "Spatial heterogeneity: MGWR" description: "Multiscale geographically weighted regression asks whether an association differs from place to place and at what spatial scale it operates: how to run it with geo.mgwr and how to read its bandwidths, coefficient maps and diagnostics." image: ../images/thumbs/heterogeneity.webp image-alt: "A panorama from the Andes to the lowlands whose light shifts from rose over the highlands to turquoise over the cloud forest" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` A regression gives one coefficient for the whole country. In 2024, across Bolivia's 343 municipalities, the Sustainable Development Index is 0.094 points higher for each percentage point of urbanization ([Regression, panel and spatial models](models.qmd)). That number is an average over very different places. **Spatial heterogeneity** means that an association is not the same everywhere, and it raises two questions: 1. **Does the association between two variables differ from place to place?** 2. **At what spatial scale does each association operate?** Is it about the same across the country, or does it change over short distances? `geo.mgwr()` answers both with multiscale geographically weighted regression (MGWR). It needs the mgwr library, which the `[mgwr]` extra installs. ```{python} #| eval: false !pip install -q "geo4bolivia[mgwr]" ``` ```{python} import pandas as pd import geo4bolivia as geo ``` ## GWR and MGWR **Geographically weighted regression** (GWR) fits a separate regression at every unit, using the units around it and giving more weight to the closer ones, so each coefficient becomes a surface that can be mapped. How far "around" reaches is the **bandwidth**, and GWR uses one bandwidth for all the terms of the model. **Multiscale GWR** (MGWR; Fotheringham, Yang & Kang 2017) finds one bandwidth *per term*. This matters because the associations in one model need not vary at the same scale: one may be nearly constant across the country while another changes from one region to the next. A single bandwidth is a compromise that smooths away the local ones and adds noise to the constant ones. With a bandwidth per term, the bandwidths themselves become a result: they say at what scale each association operates. That is why the package offers MGWR and not plain GWR. ## What `geo.mgwr` does The function follows the workflow of the mgwr library (Oshan et al. 2019): 1. **One year, complete units.** It takes one year (by default the last with data for every variable) and the units that have all the variables. Each unit is located at the centroid of its polygon. 2. **z-standardized variables.** The outcome and the regressors are rescaled to mean 0 and standard deviation 1. The coefficients are therefore in standard deviations (a coefficient of 0.5 means that one standard deviation more of the regressor goes with half a standard deviation more of the outcome) and can be compared between terms. 3. **Adaptive bandwidths, counted in nearest units.** A bandwidth of 90 means that the local regression at a unit uses the 90 units nearest to it (itself included), with weights that fall with distance and reach zero at the 90th (the default `kernel="bisquare"`). Counting neighbours instead of kilometres gives every local regression the same number of units, whether the units around it are small and close together or large and far apart. 4. **Back-fitting.** The search starts from a model with one bandwidth and then takes the terms in turn: holding the others fixed, it looks for the bandwidth of one term that gives the lowest AICc (a measure of fit that penalises complexity), and repeats the round until the estimates stop changing. This is the slow part. 5. **Local t-tests, corrected for multiple testing.** Every unit has a coefficient and a t value for every term. With hundreds of tests per term, some would pass the usual 5% level by chance, so the library divides the 5% by the term's *effective number of parameters* (a term that varies a lot over space counts as many parameters, a constant one as about one). Units that do not pass the corrected test are **grey** on the maps. ## The data The example uses the Sustainable Development Index (`index_sdg`, IDS, 0–100), the urbanization rate (`urbanization`, % of population, available for 2024 only) and multidimensional poverty (`sdg1_2_mpi`, 0–1, higher is poorer). `geo.combine()` joins the two datasets they come from. ```{python} df = geo.combine(["sdg01", "sdg"], scale="municipality", years=2024) columns = ["variable", "label", "count", "mean", "std", "min", "max"] geo.describe(df, ["index_sdg", "urbanization", "sdg1_2_mpi"])[columns] ``` ## One regressor: the index and urbanization The model has two terms: the intercept (`const`) and urbanization. Each fit on this page takes from half a minute to a minute and a half, depending on the machine. ```{python} res = geo.mgwr(df, "index_sdg", "urbanization") ``` ### Bandwidths: the scale of each association Start with the bandwidths, which are the answer to the second question. ```{python} res.gt_bw ``` ```{python} print(res.bw) print(res.bw_interval) ``` Read a bandwidth against the number of units, 343: - A bandwidth **close to the number of units** means that every local regression uses nearly all the units, so the coefficient is about the same everywhere: a **global** process. - A **small** bandwidth means that the coefficient is estimated from a unit's surroundings only and can change over short distances: a **local** process. Here the **intercept** has a bandwidth of 44 nearest units, about 13% of the municipalities, and **urbanization** has 90, about 26%. Both are far from 343, so neither is global. The level of the index and its association with urbanization both change across the country, and the association with urbanization changes more smoothly, over about twice as many neighbours, than the level does. `.bw_interval` says how sure the search is. It is the range of bandwidths that fit almost as well as the chosen one: the library builds it from the AICc of the bandwidths it tried, as the set that carries 95% of the evidence. For urbanization the interval is 86 to 113: the data do not pin down 90 exactly, but they exclude both a very local association and a global one. For the intercept it is 44 to 46. One limit to know: the search does not try very small bandwidths. In this fit the smallest it tried was 44, so read a bandwidth of 44 as "as local as the search goes": the true scale may be smaller. The other columns of the table come from the correction of the tests. **ENP** is the effective number of parameters of each term: about 22 for the intercept and 10 for urbanization, against 1 each in an ordinary regression. The **corrected alpha** is 0.05 divided by it, and the **critical |t|** is the t value a local coefficient must pass (3.08 and 2.83, instead of the usual 1.96). `.adj_alpha` and `.critical_t` hold the same numbers. ### The coefficient maps `.figs` has one map per term, by column name. This is the answer to the first question. ```{python} res.figs["urbanization"] ``` Every local coefficient is positive: more urban municipalities have a higher index everywhere. What changes is the strength, from 0.21 to 1.01 standard deviations of the index per standard deviation of urbanization. The single coefficient of an ordinary regression on the same standardized variables is 0.56 (computed below). The map of the intercept shows the other term: ```{python} res.figs["const"] ``` Because the variables are standardized, the local intercept is the level of the index, in standard deviations from the national mean, that the model gives to a municipality of average urbanization at that place. It is the part of the level that urbanization does not account for. It is significant, after the correction, in fewer than a third of the municipalities; elsewhere the local level cannot be told apart from the national mean. ### The table of local estimates `.df` has one row per unit. For the intercept (`const`) and each regressor it holds the local coefficient (`_coef`), its standard error (`_se`), its t value (`_t`) and whether it passes the corrected test (`_significant`); then the `residual` and `local_cn`. ```{python} res.df.head() ``` A summary by department puts names on the map. `geo.add_names()` adds the department of each municipality. ```{python} local = geo.add_names(res.df, groups=["department"]) by_department = local.groupby("department", as_index=False).agg( units=("geo_id", "count"), urbanization=("urbanization_coef", "mean"), significant=("urbanization_significant", "sum"), intercept=("const_coef", "mean"), intercept_significant=("const_significant", "sum"), ) by_department.sort_values("urbanization").round(2) ``` - The association with urbanization is strongest in **Potosí, Cochabamba and Chuquisaca** (department means of 0.73 to 0.79) and weakest in **Pando, Beni and Santa Cruz** (0.32 to 0.38). Where it is strongest it is more than twice as strong as where it is weakest. - It is significant in 326 of the 343 municipalities. The 17 where it is not are all in Beni (9) and Santa Cruz (8), where the coefficients are smallest. - The intercept is significant in 103 municipalities. It is positive in all 11 of Tarija (mean 0.69): at the same urbanization rate, the municipalities of Tarija have an index about 0.7 standard deviations above the national mean. In Beni the department mean is −0.45. To read the coefficients in the original units, multiply them by the ratio of the two standard deviations: ```{python} ratio = df["index_sdg"].std(ddof=0) / df["urbanization"].std(ddof=0) (local["urbanization_coef"].agg(["min", "mean", "max"]) * ratio).round(3) ``` The index is between 0.04 and 0.17 points higher per percentage point of urbanization, depending on the place, around the 0.094 of the global regression. **`local_cn`** is the local condition number, a check of local collinearity. In a local regression the regressors (and the intercept) can become nearly redundant with each other even when they are not in the whole sample, and then their local coefficients cannot be separated. Values above about 30 signal a problem, and `.notes` says how many units pass that mark. ```{python} print(local["local_cn"].describe().round(2)[["min", "50%", "max"]].to_dict()) print(res.notes) ``` The largest value is below 4, so collinearity is not a concern in this model. ### Fit: `.r2` and `.aicc` against a global model Is the local model worth its complexity? Compare it with a global regression **on the same z-standardized variables**. R² does not depend on the units, but AICc does. `geo.regression` returns the fitted statsmodels object, from which the AICc follows with the formula the mgwr library uses. ```{python} def standardized(data, columns): z = data.copy() z[columns] = (z[columns] - z[columns].mean()) / z[columns].std(ddof=0) return z def aicc(fit): # AICc of a least-squares fit, as in mgwr n, k = int(fit.nobs), len(fit.params) return -2 * fit.llf + 2 * n * (k + 1) / (n - k - 2) z = standardized(df, ["index_sdg", "urbanization"]) ols = geo.regression(z, "index_sdg", "urbanization") print("standardized coefficient:", round(ols.df["estimate"][0], 2)) pd.DataFrame({"R²": [ols.summary["r2"][0], res.r2], "AICc": [aicc(ols.models[0]), res.aicc]}, index=["least squares", "MGWR"]).round(3) ``` R² doubles, from 0.31 to 0.63, and AICc falls from 851 to 708. AICc penalises the roughly 32 effective parameters of the MGWR, so the improvement is not just the result of a more flexible model. Letting the level and the slope vary over space describes these data much better than one line for the country. `res.gt` puts the kernel, the criterion, AICc, R² and the number of units in one table. That does not make MGWR the only good description. A spatial error model has one slope for the whole country and lets the *unexplained part* be shared between neighbours: ```{python} sem = geo.spatial_model(z, "index_sdg", "urbanization", model="error") print("AIC:", round(sem.aic, 1), "| coefficient:", round(sem.df["estimate"][1], 2), "| lambda:", round(sem.lam, 2)) ``` Its AIC, 711, is close to the 708 of the MGWR (the two criteria differ slightly in how they count parameters, so the comparison is approximate). A constant slope with spatially correlated errors and a slope that varies over space fit these data about equally well, and the fit alone does not say which one is right. ### The residual map and `.interpret()` Displaying the result shows the map of the residuals (`.fig`) and a reading of the model (`.interpret()`): ```{python} res ``` The residuals are in standard deviations of the index: positive where the index is higher than the model gives. Look for groups of neighbours with the same sign, which would mean that the model still misses something that varies over space. Moran's I measures it ([Maps and spatial dependence](spatial-dependence.qmd)): ```{python} diag = geo.spatial_diagnostics(z, "index_sdg", "urbanization") left = geo.moran(res.df, "residual") print("Moran's I of the residuals. Least squares:", round(diag.moran_i_resid, 2), "| MGWR:", round(left.moran_i, 2), "(p =", left.p_sim, ")") ``` The spatial pattern in the residuals falls from 0.47 to 0.15. Most of it is gone, but not all: some clustering among neighbours remains. ## Two regressors: poverty, the index and urbanization With more than one regressor, each gets its own bandwidth, and they can differ a lot. The second model regresses multidimensional poverty on the index and urbanization. ```{python} res2 = geo.mgwr(df, "sdg1_2_mpi", ["index_sdg", "urbanization"]) res2.gt_bw ``` The three terms operate at two very different scales: - The **intercept** and the **index** have a bandwidth of 44: local processes, at the lower limit of the search. For the index, the interval is 44 to 52. - **Urbanization** has a bandwidth of 341 of 343 units, with an interval of 272 to 341: a **global** process. Its effective number of parameters is 1.4, hardly more than the single coefficient of an ordinary regression, so its tests are barely corrected (critical |t| of 2.11). ```{python} res2.figs["index_sdg"] ``` ```{python} local2 = geo.add_names(res2.df, groups=["department"]) by_department2 = local2.groupby("department", as_index=False).agg( units=("geo_id", "count"), index_sdg=("index_sdg_coef", "mean"), index_significant=("index_sdg_significant", "sum"), urbanization=("urbanization_coef", "mean"), urbanization_significant=("urbanization_significant", "sum"), ) by_department2.sort_values("index_sdg").round(3) ``` - The association between poverty and the index is **negative everywhere and local**. The coefficients run from −1.35 to −0.36: where the index is one standard deviation higher, poverty is between a third of a standard deviation and well over one standard deviation lower, depending on the place. It is strongest in Chuquisaca and Oruro (department means near −0.97) and weakest in Santa Cruz and La Paz (−0.54 and −0.58), and significant in 337 of the 343 municipalities. - The association with **urbanization is the same everywhere, and it is nil**: the local coefficients lie between 0.00 and 0.02 and none is significant. Once the index is in the model, more urban municipalities are not less poor than others, anywhere in the country. The second point is a result about scale and about size at once: a global bandwidth says that the coefficient does not vary, and here the coefficient that does not vary is close to zero. On the [models page](models.qmd), poverty on urbanization alone showed a clear negative association; it disappears once the index is in the model. ```{python} z2 = standardized(df, ["sdg1_2_mpi", "index_sdg", "urbanization"]) ols2 = geo.regression(z2, "sdg1_2_mpi", ["index_sdg", "urbanization"]) print(ols2.df[["term", "estimate", "p_value"]].round(3)) print("least squares: R² =", round(ols2.summary["r2"][0], 3), "AICc =", round(aicc(ols2.models[0]), 1)) print("MGWR: R² =", round(res2.r2, 3), "AICc =", round(res2.aicc, 1)) print("largest local condition number:", round(res2.df["local_cn"].max(), 1)) ``` The global regression agrees on the signs: −0.70 for the index, and −0.07 for urbanization, not significant at 5%. Letting the association with the index vary raises R² from 0.54 to 0.78 and lowers AICc from 713 to 555. The largest local condition number is 4.9, far below 30, so the two regressors can be told apart in every local regression. One caution about this model in particular: the multidimensional poverty index is one of the 81 indicators behind the Sustainable Development Index, and several of its components (drinking water, electricity, sanitation, school attendance) are close to other indicators of the index, such as the coverage of water, sanitation and electricity. A negative association between the two is therefore partly built in. What the MGWR adds is where that association is stronger and weaker, and that is not built in. ## Practical matters **Run time.** MGWR is slow by design: each trial bandwidth means a local regression at every unit, the search tries many bandwidths for every term, and back-fitting repeats the rounds. With the 343 municipalities, a fit with one or two regressors takes from half a minute to a minute and a half; more units and more regressors take longer. `max_iter` caps the number of back-fitting rounds (200 by default; the search stops earlier when the estimates settle). A small value, such as `max_iter=5`, gives a faster and rougher answer for a first look. Fit once, keep the result in a variable, and work with its attributes. **Few and meaningful regressors.** Each term adds a whole surface to estimate: about 10 to 22 effective parameters per local term in the examples above. With many regressors the local regressions run out of information, the regressors become collinear locally and their coefficients trade off against each other. Choose two or three regressors that you can interpret, check `local_cn`, and prefer regressors that are not strongly correlated. **Missing data.** Units with a missing value in the outcome or in any regressor are left out, and the bandwidths count neighbours among the units that remain. `res.notes` says how many were left out, and `len(res.df)` is the number of units in the model. Datasets built on an earlier municipal map (`pop`, `ntl`) have 333 units, so a model that uses them has at most 333. **Other settings.** `kernel=` changes how the weights fall with distance (`"bisquare"`, `"gaussian"` or `"exponential"`), and `criterion=` what the bandwidth search minimises (`"AICc"`, `"AIC"`, `"BIC"` or `"CV"`). `year=` picks the year. Bandwidths are always adaptive and the variables are always standardized. ## What MGWR cannot tell you - **It describes where an association differs, not why.** A coefficient that is larger in one region may reflect something about that region that the model does not include. Local coefficients absorb omitted variables that vary over space. - **It does not separate varying coefficients from spatially correlated errors.** As shown above, a global slope with a spatial error term can fit about as well. Report both. - **Local coefficients are estimated from few units.** A bandwidth of 44 means 44 weighted observations per regression. The corrected tests guard against chance findings; they do not make each local number precise. - **The coefficients are standardized against the whole country.** They are comparable between terms and places, but they are not in the units of the variables until you convert them. - **One year at a time.** The function describes a cross-section. It says nothing about how a municipality would change if its urbanization changed. - As everywhere in the package, the results are **associations**, not causes. ## In short - **GWR** fits a regression at every unit with one bandwidth for all terms; **MGWR** finds one bandwidth per term, and `geo.mgwr` is MGWR. - **`.bw`** is the scale of each association, in nearest units: near the number of units is global, small is local. **`.bw_interval`** says how sure the search is. The search does not go below about 44 units. - **`.figs`** maps the local coefficients of each term, in standard deviations, with the units that are not significant after the correction in grey. **`.df`** has the numbers, and **`local_cn`** checks local collinearity (a concern above about 30). - The index is higher where urbanization is higher in every municipality, but the association is more than twice as strong in Potosí, Cochabamba and Chuquisaca as in Pando, Beni and Santa Cruz, and it varies at a scale of about 90 municipalities. - In the poverty model, the association with the index is local (bandwidth 44) and the association with urbanization is global (341) and not significant anywhere. - Compare **`.r2`** and **`.aicc`** with a global regression on z-standardized variables, and with a spatial error model: a better fit than least squares does not rule out other spatial descriptions. - MGWR shows where associations differ and at what scale. It does not say why. ## Where next - **[Regression, panel and spatial models](models.qmd)**: the global models that MGWR is compared with. - **[Maps and spatial dependence](spatial-dependence.qmd)**: Moran's I and LISA, to see where the values themselves cluster. - **[Regional inequality](inequality.qmd)**: differences between and within departments. - **[Case study: the Sustainable Development Index](case-study-sdg.qmd)**: the index from a first map to models. - **[Function reference](../reference/index.qmd)**: every argument of `mgwr`. ## References - Fotheringham, A. S., Yang, W., & Kang, W. (2017). Multiscale geographically weighted regression (MGWR). *Annals of the American Association of Geographers*, 107(6), 1247–1265. - Oshan, T. M., Li, Z., Kang, W., Wolf, L. J., & Fotheringham, A. S. (2019). mgwr: A Python implementation of multiscale geographically weighted regression for investigating process spatial heterogeneity and scale. *ISPRS International Journal of Geo-Information*, 8(6), 269. ==================================================================================================== # Articles (https://quarcs-lab.github.io/geo4bolivia/articles/index.html) ==================================================================================================== --- title: "Articles" description: "One article per method, written around Bolivia's data, and a complete case study." toc: false listing: id: all type: grid grid-columns: 3 image-height: 170px fields: [image, title, description] sort: false contents: - case-study-sdg.qmd - data-model.qmd - exploring.qmd - spatial-dependence.qmd - convergence.qmd - transitions.qmd - inequality.qmd - models.qmd - heterogeneity.qmd - own-indicators.qmd --- Each article takes one question, shows the function that answers it on the observatory's data, and reads the result in words. Every article opens in Google Colab, where it runs without installing anything on your computer. ::: {#all} ::: ==================================================================================================== # Regional inequality (https://quarcs-lab.github.io/geo4bolivia/articles/inequality.html) ==================================================================================================== --- title: "Regional inequality" description: "How unequal Bolivia's municipalities are, whether that is changing, and whether the inequality lies between departments or inside them: Gini, Theil, the Lorenz curve and Theil's decomposition." image: ../images/thumbs/inequality.webp image-alt: "La Paz at night from the rim of El Alto: a bright dense core of lights and dim, sparse lights on the slopes" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` An inequality index puts in one number how far a set of values is from being all the same. This article asks three questions of Bolivia's municipalities: 1. **How unequal are they?** `geo.inequality()` computes the Gini index, Theil's index and the coefficient of variation, and draws the Lorenz curve. 2. **Is that changing?** With two periods the measures can be compared; with a long panel their trend can be tested. 3. **Is the inequality between departments or inside them?** `geo.theil_decomposition()` splits Theil's index in two. It uses two variables on purpose: the **Sustainable Development Index**, a score between 0 and 100, and **population**, a count. An inequality index means different things for the two, and the difference is the main thing to take from this page. ```{python} #| eval: false !pip install -q geo4bolivia ``` ## The data The dataset `sdg` has the Sustainable Development Index (`index_sdg`, IDS) and the census `population` of the 343 municipalities in two periods, stored as 2012 and 2024. ```{python} import geo4bolivia as geo import numpy as np import pandas as pd sdg = geo.load("sdg", groups=["department"]) cols = ["variable", "year", "count", "mean", "std", "min", "median", "max"] geo.describe(sdg, variables=["index_sdg", "population"])[cols] ``` The two variables could hardly be more different. The index of 2024 lies between 47.3 and 77.3, around a mean of 60.1. Population in 2024 runs from 875 people to 1.6 million, with a mean (33,134) far above the median (12,296). ::: {.callout-note} ## Inequality between municipalities, not between people Every measure on this page treats each municipality as one observation, whatever its size: the municipality of 875 people counts as much as the one of 1.6 million. The results describe how different *municipalities* are. They are not measures of inequality between people or households, which would need data on people. ::: ## 1. Three measures ```{python} res = geo.inequality(sdg, "index_sdg", measures=("gini", "theil", "cv")) res ``` All three measures are 0 when every unit has the same value and grow as the values spread out. They summarise the spread differently: | Measure | What it is | Range | Reacts most to | | --- | --- | --- | --- | | **Gini index** (`"gini"`) | the average difference between two units, over all pairs, divided by twice the mean | 0 to 1 | differences in the middle of the distribution, where most units are | | **Theil index** (`"theil"`) | the average of (value ÷ mean) × ln(value ÷ mean) | 0 to ln(number of units) | high values more than low ones; it splits exactly into groups (section 6) | | **Coefficient of variation** (`"cv"`) | the standard deviation divided by the mean | 0 upwards | the highest values, because deviations are squared | The Theil index takes logarithms, so it needs strictly positive values; with zeros in the data, use `measures=("gini", "cv")`. All three are **relative** measures: they compare the values with their mean. Doubling every value leaves them unchanged. *Adding* the same amount to every value does not: it lowers all three, because the same gaps are then smaller relative to a higher mean. Keep this in mind for section 3. `.gt` is the table of the result, and `.df` has the same numbers, one row per year: ```{python} res.gt ``` By all three measures the index is more evenly spread in 2024: the Gini index fell from 0.065 to 0.049, the Theil index from 0.0066 to 0.0038 and the coefficient of variation from 0.116 to 0.088. With two periods that is all that can be said: the values can be compared, but there is no test of a trend, which needs three or more years (`res.notes` says so). ## 2. The Lorenz curve The Lorenz curve is the picture behind the Gini index. Sort the municipalities from the lowest value to the highest. The curve shows, for each share of municipalities counted from the bottom (horizontal axis), the share of the total of the variable that they hold (vertical axis). ```{python} res.fig_lorenz ``` How to read it: - The dashed diagonal is **equality**: if every municipality had the same value, the lowest 30% of them would hold 30% of the total. - The curve can only lie on or below the diagonal. The further it sags, the more unequal the values. - The **Gini index** is the area between the diagonal and the curve, as a share of the whole triangle under the diagonal. - When two curves do not cross, the higher one is the more equal by any of the three measures. When they cross, the measures can disagree, because each weighs the parts of the distribution differently. `.lorenz_share(year, share)` reads a point off the curve: the share of the total held by the lowest `share` of units. `.lorenz` is the table behind the figure. ```{python} for year in [2012, 2024]: half = res.lorenz_share(year, 0.5) top = 1 - res.lorenz_share(year, 0.9) print(year, "| lowest half:", round(half, 3), "| highest tenth:", round(top, 3)) ``` For the index the curves almost lie on the diagonal. The half of the municipalities with the lowest index holds 45.4% of the sum of the index in 2012 and 46.5% in 2024, where equality would give 50%; the tenth with the highest index holds about 12%, where equality would give 10%. The curve of 2024 lies above the curve of 2012 along its whole length, which is why all three measures agree that the spread fell. ## 3. What an index of a score can and cannot say Now the same call on population. ```{python} pop = geo.inequality(sdg, "population", measures=("gini", "theil", "cv")) pop.fig_lorenz ``` ```{python} print(pop.df.round(3)) half, top = pop.lorenz_share(2024, 0.5), 1 - pop.lorenz_share(2024, 0.9) print("lowest half:", round(half, 3), "| highest tenth:", round(top, 3)) ``` The Gini index of population is 0.70 in both censuses. The half of the municipalities with the fewest people holds 9.8% of the population in 2024, and the tenth with the most people holds 64%. It is tempting to conclude that population is fourteen times more unequal than development (0.70 against 0.05). That comparison means nothing, for three reasons. **A count has a total that is shared out; a score does not.** For population the Lorenz curve can be read literally: 64% of the people live in a tenth of the municipalities. For the index, "the total" is the sum of 343 scores, which is not an amount of anything and which no one holds. The curve and the Gini index of a score still describe how spread out the scores are, relative to their mean, but the reading "the lowest half holds 46% of the total" is a statement about arithmetic, not about who has what. **A bounded score cannot be very unequal.** Population can differ by a factor of more than a thousand between two municipalities. An index that lies between 47 and 77 cannot: its highest value is 1.6 times its lowest. Its Gini index is bound to be small whatever happens in the country. A small value is a property of the scale, so compare the index with itself, between years or between groups of municipalities, not with other variables. **The direction of the change depends on how the score is written.** A score can be turned around: an index of what has been achieved, or an index of what is missing. The two hold the same information, but relative measures do not treat them alike. Multidimensional poverty (`sdg1_2_mpi`, from 0 to 1, lower is better) shows it clearly. It fell in every municipality between the two censuses. ```{python} mpi = geo.load("sdg01", variables="sdg1_2_mpi") mpi["not_poor"] = 1 - mpi["sdg1_2_mpi"] rows = [] for variable in ["sdg1_2_mpi", "not_poor"]: gini = geo.inequality(mpi, variable, measures="gini", lorenz=False).df spread = mpi.groupby("year")[variable].agg(["mean", "std"]).reset_index() rows.append(gini.merge(spread, on="year").assign(variable=variable)) pd.concat(rows)[["variable", "year", "mean", "std", "gini"]].round(3) ``` The Gini index of poverty **rose**, from 0.254 to 0.367, while the Gini index of its mirror image, one minus the poverty index, **fell**, from 0.078 to 0.025. Both describe the same municipalities in the same years. The standard deviation, which is identical for the two, fell from 0.106 to 0.044: the municipalities are closer to each other in index points. But mean poverty fell even faster, so the remaining gaps are larger *relative to the mean* of poverty, and smaller relative to the mean of its mirror image. Neither Gini is wrong. "Poverty became more concentrated in a few municipalities" and "municipalities became more alike in how much of their population is out of poverty" are both true. The lesson is to report the mean and the standard deviation next to any relative measure of a score, and to say which way the score runs. The Sustainable Development Index runs upwards (100 is best), and its mean rose by ten points. How much of the fall of its inequality is the higher mean? ```{python} spread = sdg.groupby("year")["index_sdg"].agg(["mean", "std"]) spread["cv"] = spread["std"] / spread["mean"] print(spread.round(3)) print(round(spread.loc[2012, "std"] / spread.loc[2024, "mean"], 3)) ``` The standard deviation fell by about a tenth, from 5.82 to 5.26 points, and the coefficient of variation by about a quarter, from 0.116 to 0.088. Had the gaps in points stayed exactly as in 2012 while the mean rose to 60.1, the coefficient of variation would still have fallen, to 0.097. So about two thirds of its fall comes from the higher level and one third from municipalities moving closer together in points. Both parts are real, and they are different facts: "everyone rose by a similar amount" and "the lowest rose more". The [convergence article](convergence.qmd) looks at the second directly. ## 4. A long panel: testing the trend With three or more years, `geo.inequality()` also estimates the **trend** of each measure: it regresses the logarithm of the measure on the year, so the slope is the average proportional change per year. The dataset `pop` has a gridded population estimate (WorldPop) for 333 municipalities in every year from 2001 to 2020. ```{python} long = geo.inequality("pop", "pop", measures=("gini", "theil", "cv")) long ``` ```{python} long.summary[["measure", "first", "last", "change", "slope", "se", "pvalue"]] ``` The Gini index of municipal population rose from 0.696 in 2001 to 0.715 in 2020. Its slope is 0.0014: the index grew by 0.14% a year on average, and the trend is statistically significant (the p-value is far below 0.01). The Theil index rose at a similar rate (0.13% a year). The coefficient of variation did not move: its slope is not distinguishable from zero (p = 0.36). So the measures disagree, and the table in section 1 says why they can. The Gini index reacts to the middle of the distribution and the coefficient of variation to its very top. Read together, they say that population became somewhat more concentrated among the municipalities in general, while the distance between the very largest ones and the average did not change. Three cautions about this result: - The series is a modelled estimate that changes smoothly from year to year, which is why the standard error of the Gini trend is so small. The test says that a straight line fits these twenty values closely; it does not have twenty independent measurements behind it. - The 333 municipalities are those whose territory did not change between the municipal maps of 2015 and 2025 (`geo.info("pop")` explains it). The result describes them. - `first=` and `last=` restrict the years, for instance `geo.inequality("pop", "pop", first=2012)`. A trend over a different window can differ. With more than four years the Lorenz figure draws only the first and the last year; `.lorenz` keeps all of them. ## 5. The spatial Gini: do neighbours differ less? The Gini index is an average over all pairs of units. `spatial=True` separates the pairs into two kinds, **neighbours** and all the others, and reports the part of the Gini index that comes from pairs of neighbours (`gini_spatial`). A permutation test then asks whether the values are arranged in space so that neighbours differ less than other pairs: it shuffles the values over the map and compares. The neighbours are those of the spatial weights, queen contiguity by default ([Maps and spatial dependence](spatial-dependence.qmd) explains them); `w=` takes others. ```{python} sp = geo.inequality(sdg, "index_sdg", spatial=True, permutations=999) linked = geo.weights("municipality").pct_nonzero print(sp.df.round(5)) print((sp.df["gini_spatial"] / sp.df["gini"]).round(4).tolist(), round(linked, 2)) ``` In 2024, pairs of neighbouring municipalities account for 0.00057 of the Gini index of 0.0493, which is 1.2% of it. That share is small mostly because few pairs are neighbours: 1.6% of all pairs of municipalities. The informative part is the comparison of the two percentages and the test. Neighbours are 1.6% of the pairs but contribute 1.2% of the inequality, so two neighbouring municipalities differ less than two municipalities taken at random, and none of the 999 shuffles gave a pattern as strong (`gini_spatial_p` = 0.001). The same holds in 2012. This is the spatial clustering of the index, seen from the side of inequality: most of the differences are between places that are far apart. It says nothing about why neighbours are alike. With `spatial=True` every measure refers to the units that have data in every year and a boundary; `res.notes` reports any that were left out. ## 6. Between and within groups Is the inequality between municipalities a matter of rich and poor *departments*, or of differences inside each department? Theil's index can be split exactly in two parts that add up to it: - **between**: the inequality there would be if every municipality had the average of its group; - **within**: the inequality among the municipalities of the same group, averaged over the groups. `by=` names the groups: a column of the table, or a coarser scale that the units nest in. ```{python} dep = geo.theil_decomposition(sdg, "index_sdg", by="department", permutations=999) dep ``` ```{python} dep.df.round(4) ``` Differences between the averages of the nine departments are 20% of the inequality between municipalities in 2012 and 15% in 2024. Most of it, 85% in 2024, is inside departments. Both parts fell, the between part faster (from 0.0014 to 0.0006) than the within part (from 0.0053 to 0.0032). ### Is the between part more than chance? Any grouping gives a between part above zero, even one drawn from a hat, because group averages always differ a little. `permutations=` tests the grouping: it reassigns the municipalities to groups of the same sizes at random, many times, and counts how often the random groups give a between part as large as the real ones (`p_between`). Here none of 999 did (p = 0.001): the departments separate municipalities with higher and lower values more than an arbitrary grouping would, although they hold only 15% of the inequality. ### More groups, a larger between share The same decomposition by province gives a very different number. ```{python} prov = geo.theil_decomposition(sdg, "index_sdg", by="province", permutations=999) prov.df.round(4) ``` Between the 113 provinces lies 66% of the inequality in 2024. That does not make provinces four times as important as departments. The between share **always grows when the groups are made finer**, for a mechanical reason: with fewer units in each group there is less left to differ inside it. At the limit, with every municipality its own group, the between share is 100%. The provinces have three municipalities each on average, and 16 of them have only one, so they contribute nothing to the within part by construction. How large is the mechanical part? Shuffle the municipalities among the groups, keeping the sizes of the groups, and see what share random groups give. This is what the permutation test does internally. ```{python} units = geo.units("municipality")[["geo_id", "department_id", "province_id"]] rng = np.random.default_rng(1) rows = [] for by, real in [("department_id", dep), ("province_id", prov)]: shares = [] for _ in range(50): shuffled = units.assign(group=rng.permutation(units[by])) table = sdg.merge(shuffled[["geo_id", "group"]], on="geo_id") random = geo.theil_decomposition(table, "index_sdg", by="group") shares.append(random.df["between_share"].iloc[-1]) rows.append({"groups": by, "number": units[by].nunique(), "between share": real.df["between_share"].iloc[-1], "random groups (mean)": np.mean(shares)}) pd.DataFrame(rows).round(3) ``` Nine random groups of the sizes of the departments hold about 2% of the inequality; the real departments hold 15%. A hundred and thirteen random groups of the sizes of the provinces hold about a third; the real provinces hold 66%. Both real groupings are well above their random counterpart, and that comparison, not the raw share, is what can be set side by side. Compare between shares only for groupings with a similar number of groups, or against the random benchmark. ### The same split for population ```{python} people = geo.theil_decomposition(sdg, "population", by="department", permutations=999) people.df.round(3) ``` For population the departments hold 6 to 7% of the inequality between municipalities, and random groupings give as much or more in most of the shuffles (p = 0.83 in 2012 and 0.69 in 2024): the between part is not distinguishable from chance. The large differences in population are between the municipalities inside each department, not between the departments' averages. ## In short - `geo.inequality()` gives the Gini index, Theil's index and the coefficient of variation in each year. They are relative measures and react to different parts of the distribution, so they can disagree, as they do for population between 2001 and 2020. - The Lorenz curve shows the whole distribution: the further it sags below the diagonal, the more unequal the values. `.lorenz_share()` reads points off it. - The Sustainable Development Index is more evenly spread across municipalities in 2024 than in 2012 by all three measures (Gini 0.065 to 0.049). About two thirds of the fall of its coefficient of variation comes from the higher mean, one third from smaller gaps in points. - An inequality index of a bounded score is small by construction, has no "total" that is shared out, and can change direction when the score is turned around. Compare it with itself, and report the mean and the standard deviation with it. - With two periods the measures can be compared; a trend test needs three or more years. - The spatial Gini shows that neighbouring municipalities differ less than others. - Differences between departments are 15% of the inequality of the index in 2024, more than random groups would give. Between shares grow with the number of groups, so compare them against a random benchmark. - The units are municipalities, each counted once. None of this measures inequality between people, and none of it explains where the differences come from. ## Where next - **[Convergence: β, σ and clubs](convergence.qmd)**: did the municipalities that started lower rise faster, and does dispersion fall over time? - **[Transitions and mobility](transitions.qmd)**: who moved up and down while the spread changed. - **[Maps and spatial dependence](spatial-dependence.qmd)**: spatial weights and the clustering behind the spatial Gini. - **[Exploring the data](exploring.qmd)**: distributions and summaries before any index. - **[Analyze](../analyze.qmd)**: all the families on one question. - Reference: [`inequality`](../reference/inequality.qmd), [`theil_decomposition`](../reference/theil_decomposition.qmd). ==================================================================================================== # Regression, panel and spatial models (https://quarcs-lab.github.io/geo4bolivia/articles/models.html) ==================================================================================================== --- title: "Regression, panel and spatial models" description: "Three kinds of regression model for subnational data, what each one compares and when it is the right tool: least squares with fixed effects and clustered errors, panel estimators with the Hausman test, and spatial models with their diagnostics and impacts." image: ../images/thumbs/models.webp image-alt: "A straight turquoise line drawn through the stars above the Cordillera Real, beside a small observatory dome" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` A regression summarises how an outcome differs between observations that differ in something else. What separates the three kinds of model on this page is *which observations are compared with which*: | Model | Function | What it compares | The right tool when | | --- | --- | --- | --- | | Least squares | `geo.regression` | Units with each other, in one year or in several pooled years; with fixed effects, only units of the same group | You want a first measure of an association and to see how it moves as controls are added | | Panel models | `geo.panel_model` | Units with each other (*between*) and each unit with itself over the years (*within*) | The same units are observed in several years, and lasting differences between units may be behind the association | | Spatial models | `geo.spatial_diagnostics`, `geo.spatial_model` | Units with each other, allowing neighbours to be related | The residuals of a regression are similar among neighbours | All three describe associations. None of them, on these data, shows what causes what. ```{python} #| eval: false !pip install -q "geo4bolivia[panel]" ``` ```{python} import numpy as np import pandas as pd import geo4bolivia as geo ``` `geo.panel_model` needs the linearmodels library, which the `[panel]` extra installs. The other functions work with the default install. ## The data The running example asks how the Sustainable Development Index (`index_sdg`, IDS, 0–100) of Bolivia's 343 municipalities relates to how urban and how populous they are. `geo.combine()` puts the variables of several datasets in one panel; multidimensional poverty (`sdg1_2_mpi`, 0–1, higher is poorer) comes along for the last section. ```{python} df = geo.combine({"sdg": ["index_sdg", "population", "urbanization"], "sdg01": ["sdg1_2_mpi"]}, scale="municipality") df["log_population"] = np.log(df["population"]) variables = ["index_sdg", "urbanization", "log_population", "sdg1_2_mpi"] columns = ["variable", "year", "count", "mean", "std", "min", "max"] geo.describe(df, variables)[columns] ``` Three things to keep in mind: - There are **two periods**, stored as 2012 and 2024. - **Urbanization exists only for 2024**, so any model that uses it is a cross-section of 2024. - Population is very unequal across municipalities, so the models use its **logarithm**, a column made here with numpy. One unit of `log_population` is a population 2.7 times larger; a doubling is 0.69. Tables and figures show a column of your own under its name, and the observatory's variables under their labels. ## Least squares: `geo.regression` ### Several specifications side by side A single regression says little about how solid an association is. The usual practice is to show several **specifications** (versions of the model) next to each other and watch what happens to the coefficient of interest as variables are added. In `geo.regression`, a list of lists is one model per list, and `fixed_effects` takes one entry per model. ```{python} res = geo.regression( df, "index_sdg", [["urbanization"], ["urbanization", "log_population"], ["urbanization", "log_population"]], fixed_effects=[None, None, "department"], ) res.gt ``` No `year` was given, so the function used the last year with data for every variable: 2024. Each column is a model; each cell is a coefficient with its standard error in parentheses below, and stars for the significance level. - **Model (1)**: the coefficient on urbanization is 0.094. A municipality whose urbanization rate is 10 percentage points higher has, on average, an IDS 0.94 points higher. For scale, the standard deviation of the IDS in 2024 is 5.3 points. The intercept, 57.6, is the IDS the line gives to a municipality with no urban population. Urbanization alone accounts for 31% of the variation in the index (R² = 0.313). - **Model (2)** adds the size of the municipality. The coefficient on urbanization moves little, to 0.086, and the coefficient on log population (0.355, standard error 0.279) cannot be told apart from zero. Among municipalities with the same urbanization rate, larger ones do not have a detectably different IDS. - **Model (3)** adds fixed effects for the nine departments. Urbanization stays at 0.098 and R² rises to 0.428. The coefficient of interest stays between 0.086 and 0.098 across the three models. A coefficient that moves a lot when a variable is added was partly picking up what that variable measures; this one does not depend on size or on the department. The parts of the result are attributes: `res.df` is the tidy table of coefficients (`model`, `term`, `estimate`, `std_error`, `p_value`, `conf_low`, `conf_high`), `res.summary` has one row per model, `res.fig` draws the coefficients with their 95% confidence intervals, and `res.models` holds the fitted statsmodels objects. ```{python} res.summary ``` ### Fixed effects: what "absorbing" means A **fixed effect** for a grouping gives every group its own intercept. `fixed_effects="department"` adds one indicator variable (a dummy) per department, so each department has its own baseline level of the IDS. The same coefficients come out if you subtract from every variable the mean of its department and run the regression on what is left. That is why the group differences are said to be **absorbed**: everything that the municipalities of a department share, measured or not, is removed before the coefficients are estimated. Two consequences: - The coefficient on urbanization in model (3) compares municipalities **of the same department** only. Whatever makes one department differ from another as a whole cannot be behind it. - A variable that is the same for all municipalities of a department cannot have a coefficient any more: the department intercepts have already taken all of its variation. The table reports the fixed effects as a Yes/No row and leaves the eight dummy coefficients out, because they are not of interest. `fixed_effects` accepts a column of your table, the name of a coarser scale (`"department"`), `"year"`, or a list of them. Fixed effects do not make the estimate causal. They rule out one family of explanations (differences between departments) and leave the others: municipalities of the same department still differ in many ways that go with both urbanization and the index. ### Clustered standard errors, and why nine clusters are too few The standard errors above are heteroskedasticity-robust (HC1, the default): they allow the size of the errors to differ between municipalities, but treat the municipalities as independent of each other. Municipalities of the same department have things in common that the model leaves out, so their errors are probably related. **Clustered** standard errors allow any correlation between the errors of units in the same cluster and assume independence only between clusters. The coefficients do not change, only their uncertainty. `cluster=` takes a column or a coarser scale. The chunk below fits model (3) with three kinds of standard error. Four provinces in different departments share the name Cercado; the package tells them apart, so `cluster="province"` gives 113 clusters, one per province. ```{python} x = ["urbanization", "log_population"] kinds = {"robust (HC1)": None, "clustered by department": "department", "clustered by province": "province"} fits = {kind: geo.regression(df, "index_sdg", x, fixed_effects="department", cluster=cluster) for kind, cluster in kinds.items()} columns = ["estimate", "std_error", "p_value", "conf_low", "conf_high"] pd.DataFrame({kind: fit.df.set_index("term").loc["urbanization", columns] for kind, fit in fits.items()}).T.round(4) ``` The estimate is 0.098 in the three rows. Clustering by department doubles the standard error, from 0.009 to 0.020, and widens the 95% interval from 0.08–0.12 to 0.05–0.15. The association is still clearly positive, but it is known much less precisely than the first table suggested. Even that wider interval should not be trusted much. The result says why: ```{python} print(fits["clustered by department"].notes) print(fits["clustered by department"].summary["std_errors"][0]) print(fits["clustered by province"].summary["std_errors"][0]) ``` The formula behind clustered standard errors measures how the errors move together inside each cluster and then treats the clusters as the independent pieces of information. It is justified when there are many clusters. With nine departments, the uncertainty is itself estimated from nine pieces, and the resulting standard errors tend to be **too small**: intervals too narrow, and tests that reject too often. The function computes p-values and intervals with a t distribution with one degree of freedom less than the number of clusters (8 here), which helps but does not solve it, and it adds a note whenever there are fewer than 30 clusters. There is no sharp threshold. What to do with few clusters: - **Cluster at a finer level** that still captures the dependence. With the 113 provinces as clusters the standard error is 0.010, close to the robust one. The formula is on firmer ground with that many clusters, but it assumes that municipalities of different provinces are unrelated, which neighbouring provinces may not be. - **Model the dependence** between neighbours directly, as the spatial models at the end of this page do. - **Report the range**. Here every version leaves the coefficient well away from zero, so the conclusion does not depend on the choice. When it does, say so. ### Pooling the years `year=` takes one year, a list of years, or `"all"`, which stacks every year with data: each municipality then appears once per year. Urbanization has one year only, so this example uses population, which has both. ```{python} pooled = geo.regression( df, "index_sdg", [["log_population"], ["log_population"]], year="all", fixed_effects=[None, "year"], cluster="geo_id", ) pooled.gt ``` There are 686 observations: 343 municipalities in two periods. - **Model (1)** treats the 686 rows as one cloud of points. A municipality 2.7 times more populous has an IDS 2.4 points higher. - **Model (2)** adds a **year fixed effect**, an intercept for each period. The mean IDS rose from 50.1 to 60.1 between the two periods; the year effect absorbs that rise, so the coefficient (2.1) compares municipalities within the same period. R² jumps from 0.13 to 0.55, and almost all of that jump is the common rise, not population. The coefficient on log population is about 2 here and was indistinguishable from zero in the 2024 table above. The difference is urbanization: larger municipalities are more urban (in 2024 the correlation between log population and the urbanization rate is 0.62), and once urbanization is in the model, size adds nothing. Urbanization is not available for 2012, so it cannot be added to the pooled model. Two rows of the same municipality are not independent observations. `cluster="geo_id"` allows for that. If you pool without it, the result reminds you: ```{python} geo.regression(df, "index_sdg", "log_population", year="all").notes ``` Pooled least squares mixes two comparisons: between municipalities, and between the two periods of the same municipality. Panel models separate them. ## Panel models: `geo.panel_model` A **panel** observes the same units in several years. `geo.panel_model` fits the four standard estimators side by side. They use the same data and differ in what they compare: | Estimator | `models=` | What it compares | The question it answers | | --- | --- | --- | --- | | Pooled least squares | `"pooled"` | All rows, as one cloud | A mix of the two below | | Between | `"between"` | The units' averages over the years, one row per unit | Do units with a higher regressor have a higher outcome than other units? | | Fixed effects (within) | `"fe"` | Each unit's deviations from its own average | When a unit's regressor is above its own average, is its outcome too? | | Random effects | `"re"` | A weighted mix of between and within | The same as fixed effects, more precisely, *if* its assumption holds | The fixed-effects estimator gives every unit its own intercept. As with the department effects above, this absorbs every lasting difference between units, measured or not: whatever does not change over the years cannot be behind a within estimate. The price is that it uses only the variation over time, which is often small. Random effects treat the unit intercepts as random draws that are **unrelated to the regressors**. When that holds, they use both sources of variation and are more precise. When it does not, they are pulled towards the between comparison, lasting differences included. The **Hausman test** checks it by comparing the fixed- and random-effects coefficients: if they differ by more than chance allows (a small p-value), the assumption fails and fixed effects are the estimates to read. ### A long panel: night-time lights and population density The SDG data have two periods. For a longer panel, `geo.combine()` joins night-time lights per capita (`ln_NTLpc`, in logs, 2012–2020) and population density (`pop_density`, people per km²). Both are computed on an earlier municipal map and cover 333 municipalities. ```{python} panel = geo.combine(["ntl", "pop"], scale="municipality") d = geo.describe(panel, ["ln_NTLpc", "pop_density"]) d[d["year"].isin([2012, 2020])][["variable", "year", "count", "mean", "median"]] ``` Displaying the result shows the coefficients of the four estimators with their confidence intervals, and a reading of them: ```{python} pm = geo.panel_model(panel, "ln_NTLpc", "pop_density") pm ``` ```{python} pm.gt ``` The model uses the rows with both variables: 2,930 observations of 331 municipalities over nine years. - **Between** (0.0012): municipalities that are denser on average over the nine years have more light per person. A difference of 100 people per km² goes with 0.12 more log points of lights per capita, about 13% more light per person. - **Within** (0.0022): in the years when a municipality is denser than its own average, it also has more light per person than its own average. - The **Hausman test** (χ² = 2.34 with 1 degree of freedom, p = 0.13) does not reject random effects, whose estimate (0.0014) lies between the other two. - Density accounts for very little of the variation in either direction: the between R² is 0.05 and the within R² is below 0.01. Standard errors are clustered by municipality by default (`cluster=True`), which allows the errors of the same municipality to be related across years. `.summary` and `.hausman` hold the numbers: ```{python} print(pm.hausman) pm.summary[["estimator", "n", "units", "years", "r2_within", "r2_between"]] ``` ### Year effects change the within estimate Between 2012 and 2020 both variables rose on average: the table above shows the mean of log lights per capita going from 13.1 to 13.9, and mean density from 52 to 63 people per km². A within estimator compares each municipality with itself over the years, so a rise that all municipalities share counts as association, even if the two rises have nothing to do with each other. `time_effects=True` adds an intercept for each year, which removes whatever is common to all units in a year. ```{python} pm_t = geo.panel_model(panel, "ln_NTLpc", "pop_density", time_effects=True) pm_t.gt ``` The within estimate changes sign, from 0.0022 to −0.0019. The question it now answers is narrower: in the years when a municipality's density is higher than its own average *by more than the country's movement that year*, are its lights per capita higher too? The answer is no: they are lower. The between estimate is the same as before, because averaging over the years leaves no year to adjust for. Two cautions about this result: - **The negative within estimate is partly built in.** Lights per capita are lights divided by population, and density is population divided by an area that does not change. Within a municipality, density rises only when population rises, and if lights do not keep pace, lights per capita fall. A variable that sits in the numerator of one side and the denominator of the other produces a negative association by construction. - **The Hausman test now rejects random effects** (χ² = 38.7, p < 0.001), where it did not before. The test is about a specification, not about the data in general. Decide whether the model needs year effects first, on the grounds of what could be common to all units, and read the test after. ### Two periods: the SDG data With two periods the same issue is sharper. Here are the four estimators for the IDS and log population: ```{python} two = geo.panel_model(df, "index_sdg", "log_population") two.gt ``` Between municipalities, a population 2.7 times larger goes with an IDS 2.1 points higher. The fixed-effects estimate is 19.4, nine times that. It is not a finding about population. With two periods, the fixed-effects estimate is the regression of each unit's *change* in the outcome on its *change* in the regressor. Without year effects, that regression has no intercept, so the line is forced through the origin. `geo.change()` shows what the changes look like: ```{python} chg_ids = geo.change(df, "index_sdg")["index_sdg_change"] chg_pop = geo.change(df, "log_population")["log_population_change"] print("IDS rose in", (chg_ids > 0).sum(), "of", len(chg_ids), "municipalities;", "mean change", round(chg_ids.mean(), 1)) print("population grew in", (chg_pop > 0).sum(), "; mean change of the log", round(chg_pop.mean(), 2)) ``` The index rose by about 10 points almost everywhere, and population grew by a small amount in most places. A line through the origin can only account for a rise of 10 points that goes with a rise of 0.13 in log population by being very steep. The estimator attributes a change common to all municipalities to the regressor. ```{python} two_t = geo.panel_model(df, "index_sdg", "log_population", time_effects=True) two_t.gt ``` With `time_effects=True` the common rise goes to the year effect (the intercept of the regression of changes), and the fixed-effects estimate becomes −0.4 with a standard error of 0.9: municipalities whose population grew more did not see their IDS rise more, or less, than the others. The between association (2.1) remains: larger municipalities have a higher index, but growing did not go with improving faster. `.interpret()` warns about this whenever a fixed-effects model has two years and no year effects. With two periods, use `time_effects=True` unless you have a reason to believe that nothing changed for all units at once. ## Spatial models ### Why related residuals are a warning Least squares assumes that what the model leaves unexplained in one unit (its **residual**) says nothing about the residual of another. On a map this often fails: neighbouring municipalities share conditions that the model does not include, so their residuals are alike. Residuals that are similar among neighbours are **spatially autocorrelated**, and it matters for two reasons: - The model has **less independent information** than its number of rows suggests, so its standard errors are too small. - The model may be **missing a spatial process**. If the outcome of a municipality is related to the outcomes of its neighbours, leaving that out can bias the coefficients, not only their standard errors. `geo.spatial_diagnostics()` fits the least-squares model and tests its residuals. Like every spatial function of the package, it needs to know who is a neighbour of whom: by default, municipalities that share a border (queen contiguity), each neighbour weighted equally. [Maps and spatial dependence](spatial-dependence.qmd) explains the weights and Moran's I. ```{python} diag = geo.spatial_diagnostics(df, "index_sdg", "urbanization") diag.gt ``` **Moran's I of the residuals** is 0.47 and far from what chance would give: after accounting for urbanization, municipalities with an index above the line are surrounded by others above the line. The standard errors of the first table of this page are too optimistic, and a spatial model is worth fitting. ### The Lagrange multiplier tests and the model they point to The other rows are **Lagrange multiplier (LM) tests**. Each one asks, using only the least-squares fit, whether adding one kind of spatial term would improve the model: - **LM lag**: a term for the neighbours' *outcome*. The alternative is the **spatial lag model**, in which the IDS of a municipality is related to the IDS of its neighbours. - **LM error**: a term for the neighbours' *errors*. The alternative is the **spatial error model**, in which what the model leaves out is spatially correlated but the outcome does not depend on the neighbours' outcome. The two simple tests react to either kind of dependence, so both usually reject together, as here (168.5 and 187.1). The **robust** versions test one kind while allowing for the other, and they decide: ```{python} print(diag.recommendation) print(diag.reasoning) ``` The rule is: if neither simple test rejects, keep least squares; otherwise read the robust tests, and choose the model whose robust test rejects, or the one with the larger statistic when both do. (If neither robust test rejects, the larger simple statistic gives a tentative answer.) Here the robust LM error (23.1) is much larger than the robust LM lag (4.5), so the tests point to the **error model**. The robust lag test also rejects at 5% (p = 0.03), which is why the function adds that a Durbin model is worth a look. ### Lag, error and Durbin models Writing $W y$ for the average outcome of a unit's neighbours and $W x$ for the average of their regressor, the three main models are: | Model | `model=` | Equation | What is spatial | | --- | --- | --- | --- | | Spatial lag | `"lag"` | $y = \rho W y + \beta x + \varepsilon$ | The outcome depends on the neighbours' outcome | | Spatial error | `"error"` | $y = \beta x + u,\; u = \lambda W u + \varepsilon$ | The unexplained part is shared with neighbours | | Spatial Durbin | `"durbin"` | $y = \rho W y + \beta x + \theta W x + \varepsilon$ | The neighbours' outcome and the neighbours' regressor | The Durbin model contains the lag model (when θ = 0), and it can also reproduce the error model, so it is the cautious choice when the tests do not separate the two. `geo.spatial_model` also fits `"slx"` (the neighbours' regressor only), `"durbin_error"` and `"ols"`, the last one for comparison. ```{python} kinds = ["ols", "lag", "error", "durbin"] sp = {kind: geo.spatial_model(df, "index_sdg", "urbanization", model=kind) for kind in kinds} coef = lambda fit: fit.df.set_index("variable").loc["urbanization", "estimate"] pd.DataFrame({kind: {"coefficient on urbanization": coef(fit), "rho": fit.rho, "lambda": fit.lam, "AIC": fit.aic, "pseudo R²": fit.r2} for kind, fit in sp.items()}).T.round(3) ``` - The spatial parameters are large: ρ = 0.62 in the lag model, λ = 0.67 in the error model. Neighbouring municipalities resemble each other well beyond what urbanization accounts for. - **AIC** (lower is better, on the same data) falls from 1,987 for least squares to about 1,850 for the spatial models. The error and Durbin models are practically tied (1,849.4 and 1,849.0) and both are ahead of the lag model (1,854.6), in line with the tests. - Compare these models by AIC, not by `.r2`. It is a pseudo-R² that is not comparable between them: the error model's is the same 0.313 as least squares, because it counts only what the regressor accounts for. The error model, which the tests recommend, is the simplest to read. Its coefficient has the same meaning as in least squares: ```{python} sp["error"] ``` A municipality 10 points more urban has an IDS about 0.8 points higher, against 0.94 in least squares, and the standard error now allows for the dependence between neighbours. ### Direct, indirect and total impacts In a lag or Durbin model **a coefficient is not the whole association**. The outcome of each unit is tied to its neighbours' outcomes, which are tied to their own regressors and to *their* neighbours, and so on back to the unit itself. The model therefore implies three quantities for each regressor, called **impacts**: - **Direct**: how the outcome of a unit differs with its own regressor, including the part that comes back through its neighbours. - **Indirect** (often called spillover): how the outcome of a unit differs with the regressor of the *other* units, summed over them. - **Total**: the sum of the two. ```{python} impacts = {kind: sp[kind].impacts.drop(columns="term") for kind in ["lag", "durbin"]} pd.concat(impacts).droplevel(1).round(3) ``` In the lag model the coefficient on urbanization is 0.072, but the total impact is 0.189, more than twice as much: it equals β / (1 − ρ) = 0.072 / (1 − 0.618). Reading 0.072 as "the association with urbanization" would understate what the model says. The two models disagree about the indirect part. The lag model puts it at 0.109, larger than the direct impact. This is a property of the model as much as of the data: with a single ρ, the ratio of indirect to direct impact is the same for every regressor and is set by ρ and the weights alone. The Durbin model lets the neighbours' urbanization have its own coefficient, and gives an indirect impact of 0.051 with a standard error of 0.030, less than twice its standard error. The error model has no indirect impact by construction. The standard errors of the impacts come from simulation (`n_draws=10_000` draws by default; `seed=` makes them reproducible). What can be said from the three models together: - The **direct association** is stable: between 0.08 and 0.09 IDS points per percentage point of urbanization in every model. - **Whether a municipality's index also goes with the urbanization of its neighbours is not settled** by these data: the answer runs from zero (error model) through an imprecise 0.05 (Durbin) to 0.11 (lag), and the tests and the AIC favour the models that give the smaller answers. - "Spillover" is the name of a term in the model. It measures an association with the neighbours' values, and it does not show that urbanization in one municipality changes anything in another. The results also depend on who counts as a neighbour. A quick check with other weights, the six nearest municipalities instead of those sharing a border: ```{python} w6 = geo.weights("municipality", method="knn", k=6) diag6 = geo.spatial_diagnostics(df, "index_sdg", "urbanization", w=w6) error6 = geo.spatial_model(df, "index_sdg", "urbanization", model="error", w=w6) print(diag6.recommendation, "| coefficient:", round(coef(error6), 3), "| lambda:", round(error6.lam, 2)) ``` The tests point to the error model again, and its coefficient (0.083) is close to the 0.081 found with shared borders. ### Another outcome, another model The tests do not always point the same way. For multidimensional poverty on urbanization they choose the lag model: ```{python} diag_mpi = geo.spatial_diagnostics(df, "sdg1_2_mpi", "urbanization") print(diag_mpi.recommendation) diag_mpi.df.round(4) ``` Here the robust LM lag rejects (9.1, p = 0.003) and the robust LM error does not (1.5, p = 0.23). ```{python} lag_mpi = geo.spatial_model(df, "sdg1_2_mpi", "urbanization", model="lag") lag_mpi ``` ```{python} lag_mpi.impacts.round(5) ``` Poverty is lower where urbanization is higher, and the poverty of neighbouring municipalities moves together (ρ = 0.59). A municipality 10 points more urban has a poverty index 0.005 lower directly and 0.011 lower in total, against a mean of 0.063 in 2024. As before, the indirect part is what a lag model implies once ρ is estimated, and it should be read with the same caution. Both functions take `fixed_effects=` (a column or a coarser scale, added as dummies), `w=` for other weights, and `year=`; they analyse one year at a time. `geo.spatial_model` estimates by maximum likelihood by default (`method="ml"`) and by the generalized method of moments with `method="gm"`. ## In short - **`geo.regression`** puts specifications side by side. The IDS is 0.09 to 0.10 points higher per percentage point of urbanization in 2024, with or without population and department fixed effects. - **Fixed effects** absorb everything the units of a group share, so the coefficients compare units of the same group only. They rule out one family of explanations, not all. - **Clustered standard errors** allow for related errors inside a cluster. With nine departments they are unreliable and tend to be too small; cluster at a finer level, model the spatial dependence, or report the range. - **`year="all"`** pools the years; add `"year"` fixed effects for what changed everywhere and `cluster="geo_id"` for the repeated units. - **`geo.panel_model`** separates the between comparison (units with each other) from the within comparison (each unit with itself). The Hausman test says whether random effects are admissible for a given specification. - **Without year effects, a within estimate also absorbs changes common to all units.** It turned a null association between population growth and the change in the IDS into a coefficient of 19, and it changed the sign of the lights and density estimate. With two periods, use `time_effects=True`. - **`geo.spatial_diagnostics`** tests the residuals of least squares for spatial dependence and points to a lag or an error model; **`geo.spatial_model`** fits it. In lag and Durbin models, read the direct, indirect and total impacts, not the coefficients. - Every number on this page is an association between municipalities. None of the models identifies a cause. ## Where next - **[Maps and spatial dependence](spatial-dependence.qmd)**: spatial weights, Moran's I and LISA, the tools behind the spatial models. - **[Spatial heterogeneity: MGWR](heterogeneity.qmd)**: when the question is whether a coefficient differs from place to place. - **[Convergence: β, σ and clubs](convergence.qmd)**: a regression of growth on the starting level, with the same spatial estimators. - **[Case study: the Sustainable Development Index](case-study-sdg.qmd)**: these models as steps of a complete analysis. - **[Function reference](../reference/index.qmd)**: every argument of `regression`, `panel_model`, `spatial_diagnostics` and `spatial_model`. ==================================================================================================== # Building your own indicators (https://quarcs-lab.github.io/geo4bolivia/articles/own-indicators.html) ==================================================================================================== --- title: "Building your own indicators" description: "From a table or a raster of your own to an analysis: link it to the geo IDs, compute zonal statistics, move it across scales, save it and analyse it." image: ../images/thumbs/own-indicators.webp image-alt: "An Andean loom with threads in turquoise, rose and sand joining into one woven pattern" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` The observatory's datasets are a starting point. Sooner or later you will have data of your own: a table from a ministry or a survey, or a raster (a gridded image, such as a satellite product). This article takes both to the point where every function of the package works on them: | Step | Function | What it does | | --- | --- | --- | | 1 | `geo.link`, `geo.check` | attach geo IDs to a table that names its units by name or code, and check the result | | 2 | `geo.zonal` | summarise a raster over the units of a scale | | 3 | `geo.combine` | put your indicator and the observatory's datasets in one panel | | 4 | `geo.aggregate` | move a panel up the hierarchy, with a rule per variable | | 5 | `geo.overlay` | move it onto units that do not nest, such as a grid | | 6 | `geo.save` | write it as a documented dataset | | 7 | any analysis function | analyse it like any other dataset | The only requirement of everything that follows is a column called `geo_id` ([The data model](../articles/data-model.qmd) explains the IDs). ```{python} #| eval: false !pip install -q "geo4bolivia[raster]" ``` The `[raster]` extra adds rasterio, which `geo.zonal()` needs. The files this page writes go to a temporary folder, `work`. ```{python} import tempfile from pathlib import Path import geopandas as gpd import numpy as np import pandas as pd import geo4bolivia as geo work = Path(tempfile.mkdtemp()) ``` ## 1. A table of your own ### Link by name where names are unique Tables from reports identify units by name, in every possible spelling. `geo.link()` adds the `geo_id`, ignoring accents, capitals, spaces and punctuation. The numbers in the tables of this section are **made up** for the demonstration. ```{python} by_department = pd.DataFrame({ "Departamento": ["LA PAZ", "Cochabamba", "Santa Cruz", "Potosi", "chuquisaca"], "year": 2024, "clinics": [412, 301, 388, 266, 190], }) geo.link(by_department, scale="department", on="Departamento") ``` The nine department names are unique, so this is safe. ### Why names fail for municipalities Municipality names are not unique: eight names are shared by two or three municipalities in different departments. A typing mistake is the other usual problem. `geo.link()` does not guess in either case. With `errors="warn"` it links what it can, leaves the rest without a `geo_id` and explains each problem (the default, `errors="raise"`, stops with the same explanation): ```{python} import warnings by_name = pd.DataFrame({"municipio": ["Sucre", "El Alto", "San Pedro", "Cochabmba"], "clinics": [41, 63, 5, 30]}) with warnings.catch_warnings(record=True) as problems: warnings.simplefilter("always") linked = geo.link(by_name, scale="municipality", on="municipio", errors="warn") for problem in problems: print(problem.message, "\n") linked ``` "San Pedro" is the name of three municipalities, in Pando, Potosí and Santa Cruz, and the table does not say which one is meant. A match on the name could attach the row to the wrong one without any warning, which is worse than an error. ### Link by code Codes do not have this problem. Official statistics carry the municipality code of the National Statistics Institute (INE), and `key=` names the column of `geo.units()` to match: `"mun_id"` for the code as a number, `"ine_code"` for the code as text with its leading zero (`"020101"`). ```{python} by_code = pd.DataFrame({"codigo": [10101, 20105, 20101, 50501, 71005], "year": 2024, "clinics": [41, 63, 58, 5, 9]}) clinics = geo.link(by_code, scale="municipality", on="codigo", key="mun_id") geo.add_names(clinics, groups=["department"]) ``` The last two rows are two different municipalities called San Pedro, each linked to its own ID. Remember that a geo ID is not an INE code: La Paz is `BO-04-17-04` here and `020101` for INE. ### Check the result `geo.check()` lists what would stop a table from being a valid panel: unknown IDs, more than one row per unit and year, years that are not integers. An empty list means the table is ready. ```{python} print(geo.check(clinics, scale="municipality")) broken = pd.DataFrame({"geo_id": ["BO-04-17-04", "BO-04-17-04", "BO-99-01-01"], "year": [2024, 2024, 2024], "clinics": [58, 60, 7]}) geo.check(broken, scale="municipality") ``` ## 2. A raster of your own `geo.zonal()` summarises a raster over the units of a scale: for each municipality, the sum, mean, minimum, maximum, median or count of the pixels whose centre falls inside it. It reads any GeoTIFF, from a path or a URL: an export from Google Earth Engine, a population grid, a land-cover map. So that this page needs no download, we make a small raster ourselves: the straight-line distance, in kilometres, from every pixel to the nearest of the three largest cities (La Paz, Cochabamba and Santa Cruz de la Sierra). It is a crude measure of remoteness, good enough to show the mechanics. With your own GeoTIFF, skip this chunk. ```{python} import rasterio from rasterio.transform import from_bounds CITIES = {"La Paz": (-16.4955, -68.1336), "Cochabamba": (-17.3895, -66.1568), "Santa Cruz de la Sierra": (-17.7833, -63.1821)} # latitude, longitude x0, y0, x1, y1 = geo.boundaries("country").total_bounds n = 600 # pixels per side lon, lat = np.meshgrid(np.linspace(x0, x1, n), np.linspace(y1, y0, n)) def km_to(lat0, lon0): """Approximate distance in km from every pixel to one point.""" return 111.2 * np.hypot((lon - lon0) * np.cos(np.radians(lat0)), lat - lat0) km = np.min([km_to(*city) for city in CITIES.values()], axis=0) tif = work / "distance.tif" with rasterio.open(tif, "w", driver="GTiff", width=n, height=n, count=1, dtype="float32", crs="EPSG:4326", transform=from_bounds(x0, y0, x1, y1, n, n)) as file: file.write(km.astype("float32"), 1) ``` One call turns the raster into a panel. `name=` is the prefix of the new columns and `year=` the year to record. ```{python} access = geo.zonal(tif, scale="municipality", stats=["mean", "max"], name="dist_km", year=2024) print(geo.check(access)) access.head(3) ``` ```{python} geo.map(access, "dist_km_mean", scheme="quantiles", title="Mean distance to the nearest main city (km)") ``` The table has the same shape as a dataset loaded with `geo.load()`, so `geo.map()` draws it without being told anything else. The mean distance ranges from 8 km to 763 km across the 343 municipalities. Three things to know about `zonal`: - **Pixel size against unit size.** A pixel counts when its centre is inside the unit, and pixels are not split at the edges. The pixels here are a little over 2 km on each side and the smallest municipality has 13 km²; no municipality came out empty (`access` has no missing values). With a coarser raster, small units can: use a finer raster or `all_touched=True`, which counts every pixel a unit touches. - **Sum or mean.** Use `sum` for rasters of counts (people per pixel, radiance) and `mean` for rasters of rates or intensities (temperature, distance, a share). - **A series.** Pass `{2019: "a.tif", 2020: "b.tif"}` instead of one path to get one row per unit and year. ## 3. Next to the observatory's data `geo.combine()` merges datasets into one panel at one scale. A dictionary picks the variables of each dataset, and `extra=` takes your own panels: ```{python} panel = geo.combine({"sdg": ["population"], "sdg01": ["sdg1_2_mpi"]}, scale="municipality", years=2024, extra=[access]) panel.head(3) ``` From here a new indicator is one line of pandas. Columns of two datasets with the same name get the dataset id as a prefix, so nothing is overwritten silently. ## 4. Up the hierarchy `geo.aggregate()` moves a panel to a coarser scale. For the observatory's variables it takes the **aggregation rule** from the metadata (`dataset=`). Your own variables have no metadata yet, so you state the rule, and thinking about it is the point: there is no single right way to turn 87 municipal distances into one number for La Paz. | Rule | Meaning | Here | | --- | --- | --- | | `sum` | add the units | `population` | | `mean` | the average unit, each counting the same | the distance of the average *municipality* | | `weighted_mean:` | an average weighted by another column | the distance of the average *person* | | `max`, `min` | the extreme unit | the most remote place of the department | ```{python} rules = {"population": "sum", "dist_km_mean": "weighted_mean:population", "dist_km_max": "max"} by_person = geo.aggregate(panel, to="department", rules=rules, variables=list(rules)) by_unit = geo.aggregate(panel, to="department", rules={"dist_km_mean": "mean"}, variables="dist_km_mean") by_person["dist_km_simple"] = by_unit["dist_km_mean"] geo.add_names(by_person).round(0) ``` The two means answer different questions. In La Paz the average municipality is 103 km from a main city and the average person 56 km, because more than half of the department's people live in two municipalities, El Alto and Nuestra Señora de La Paz. `n_units` and `coverage` say how many municipalities each value rests on, and what share of the department's municipalities that is. ### Rates and logs: aggregate the totals Some variables cannot be aggregated as they are. Night-time lights are published as the logarithm of lights per person (`ln_NTLpc`), whose rule in the metadata is `none`: the mean of the municipal logarithms is not the logarithm of the department's lights per person. The way through is to go back to totals, add them up, and build the ratio again. ```{python} lights = geo.combine(["ntl", "pop"], scale="municipality", years=2020) lights = lights.dropna(subset=["ln_NTLpc"]) lights["ntl"] = np.exp(lights["ln_NTLpc"]) * lights["pop"] # total lights dep = geo.aggregate(lights, to="department", rules={"ntl": "sum", "pop": "sum"}, variables=["ntl", "pop"]) dep["ln_NTLpc"] = np.log(dep["ntl"] / dep["pop"]) geo.add_names(dep)[["geo_id", "name", "ln_NTLpc", "n_units", "coverage"]].round(2) ``` `coverage` is below 1 in four departments: night-time lights exist for 333 of the 343 municipalities, and some of those have no value in 2020 (the `dropna` line removes them, so that their population is not counted without their lights). A department total built from part of its municipalities should be reported with that share. ## 5. Units that do not nest Departments are unions of municipalities, so aggregation is exact. A grid, a set of watersheds or another vintage of the municipal map cut across municipalities. `geo.overlay()` moves data onto such units by **area weighting**. You say which variables are *extensive* (totals: each municipality's total is split among the cells in proportion to the area that falls in each) and which are *intensive* (rates: each cell gets the area-weighted mean of the municipalities it covers). First a grid of 50 km cells, built in the project's equal-area projection so that the cells have the same size everywhere: ```{python} from shapely.geometry import box crs = geo.config()["equal_area_crs"] # metres, equal area gx0, gy0, gx1, gy1 = geo.boundaries("country").to_crs(crs).total_bounds size = 50_000 cells = [box(x, y, x + size, y + size) for x in np.arange(gx0, gx1, size) for y in np.arange(gy0, gy1, size)] grid = gpd.GeoDataFrame({"geo_id": [f"cell-{i:03d}" for i in range(len(cells))]}, geometry=cells, crs=crs).to_crs("EPSG:4326") ``` ```{python} on_grid = geo.overlay(panel, grid, extensive=["population"], intensive=["sdg1_2_mpi"]) print(len(on_grid), "of", len(grid), "cells touch the country") print(f"{panel['population'].sum():,.0f} people in the municipalities,", f"{on_grid['population'].sum():,.0f} on the grid") ``` The total is kept: no person is lost or counted twice. `geo.map()` also draws tables that bring their own geometry: ```{python} geo.map(grid.merge(on_grid), "population", scheme="quantiles", title="Census population 2024 on a 50 km grid") ``` ::: {.callout-warning} ## What area weighting assumes `overlay` spreads each municipality's value evenly over its territory. People are not spread evenly: they live in towns. The grid above therefore shows where the populous *municipalities* are, not where people are within them, and this matters more the larger the municipalities are compared with the cells. For a variable that has a raster source, `geo.zonal()` on the grid is the better tool (it accepts a GeoDataFrame as `scale`). ::: ## 6. Save it as a documented dataset `geo.save()` writes a panel the way the observatory publishes its own datasets: a CSV file, a Parquet file with the same content, and a metadata file that describes every variable. The `variables` dictionary is where you record what you know now and will have forgotten in six months: a label, how the variable was built, its unit, its source and its aggregation rule. ```{python} package = geo.save( access, "access", scale="municipality", directory=work / "my_data", title="Distance to the main cities", description="Straight-line distance from each municipality to the nearest of " "La Paz, Cochabamba and Santa Cruz de la Sierra.", variables={ "dist_km_mean": {"label": "Mean distance to a main city (km)", "description": "Mean over the municipality's pixels", "unit": "km", "source": "Own calculation", "aggregation": "weighted_mean:population"}, "dist_km_max": {"label": "Maximum distance to a main city (km)", "description": "Maximum over the municipality's pixels", "unit": "km", "source": "Own calculation", "aggregation": "max"}}, sources=["Own calculation from the boundaries of SDSN Bolivia (2025)"]) sorted(str(path.relative_to(work)) for path in (work / "my_data").rglob("*.*")) ``` The function returns the metadata it wrote. Each variable is a field with the properties the [data model](../articles/data-model.qmd) describes: ```{python} package["resources"][0]["schema"]["fields"][2] ``` The folder has the layout of the data repository (`data//.csv`, `metadata/.datapackage.json`), so it can be shared as it is, and read back with `pd.read_csv()`. To propose a dataset for the observatory itself, it must also be reproducible from its original files; the data repository's [contribution guide](https://github.com/quarcs-lab/gdo4bolivia/blob/main/CONTRIBUTING_DATA.md) gives the steps, and the metadata file written here is a starting point for it. ## 7. Analyse it Nothing more is needed. Every analysis function takes a table with `geo_id`, finds the boundaries and builds the spatial weights. Is the new indicator clustered in space? Global Moran's I measures whether neighbouring units have similar values (0: no more than by chance; close to 1: very similar). A column of your own has no label in the metadata, so figures and readings would show its name. `geo.labels(table, own=...)` gives it one: ```{python} geo.labels(panel, own={"dist_km_mean": "Mean distance to a main city (km)"}) res = geo.moran(panel, "dist_km_mean") res ``` Moran's I is 0.94. That is no discovery: distance changes smoothly over space by construction, so neighbours must be alike. It is a useful check all the same, because it shows that the table, the boundaries and the weights fit together, and it is a reminder that a strongly clustered variable will correlate with many other clustered variables for no deeper reason. Is remoteness, measured this way, associated with poverty? ```{python} geo.scatter(panel, "dist_km_mean", "sdg1_2_mpi", title="Poverty and distance to the nearest main city, 2024") ``` Hardly: across the 343 municipalities the correlation is 0.04 and the line accounts for none of the variation, so there is no linear association between the two in 2024. This is one crude measure of remoteness and one year; it does not show that location is irrelevant to poverty. The analysis functions are not limited to the observatory's units either. For the grid, pass its polygons as `gdf=`: ```{python} on_cells = geo.moran(on_grid, "population", gdf=grid) print(round(on_cells.moran_i, 2), on_cells.p_sim, "|", on_cells.w_spec) ``` On the 501 cells with data, population is positively but weakly clustered (I = 0.19), with neighbours defined among the cells themselves. ## In short - Everything works on a table with a `geo_id` column. `geo.link()` adds it to your table and `geo.check()` verifies the result. - Link municipalities **by code** (`key="mun_id"` or `"ine_code"`), not by name: eight names are shared by several municipalities, and `geo.link()` refuses to guess. - `geo.zonal()` turns a raster into a panel; mind the pixel size, and choose `sum` for counts and `mean` for rates. - `geo.combine()` puts your indicator next to the observatory's data. `geo.aggregate()` needs a rule per variable; for rates and logs, aggregate the totals and rebuild the ratio. - `geo.overlay()` reaches units that do not nest, under the assumption that values are spread evenly within each unit. - `geo.save()` writes the data with their documentation. After that, your indicator is a dataset like any other. ## Where next - **[The data model](../articles/data-model.qmd)**: geo IDs, source codes, aggregation rules and metadata in detail. - **[Exploring the data](../articles/exploring.qmd)**: a first look at any variable, yours included. - **[Maps and spatial dependence](../articles/spatial-dependence.qmd)**: Moran's I and local clusters, explained. - **[Regression, panel and spatial models](../articles/models.qmd)**: use your indicator as an explanatory variable. - **[Function reference](../reference/index.qmd)**: `link`, `check`, `zonal`, `combine`, `aggregate`, `overlay` and `save`. ==================================================================================================== # Maps and spatial dependence (https://quarcs-lab.github.io/geo4bolivia/articles/spatial-dependence.html) ==================================================================================================== --- title: "Maps and spatial dependence" description: "From a map to a test: classed maps, spatial weights, global Moran's I and LISA clusters of multidimensional poverty in Bolivia's municipalities, and how far the conclusions depend on the choices made." image: ../images/thumbs/spatial-dependence.webp image-alt: "The Altiplano at night with village lights gathered in a rose cluster and a turquoise cluster, a volcano behind" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` A map of poverty almost always shows patches: poor municipalities next to poor ones, better-off ones next to each other. The eye is quick to see patches, also where there are none, and the patches it sees depend on how the map was coloured. This article goes from the map to a test of what it seems to show, in four steps: 1. **Draw the map**, and see how much the choice of classes changes it. 2. **Define "neighbour"** with spatial weights, and inspect the result. 3. **Test the whole map** with global Moran's I: are similar values closer together than chance would put them? 4. **Locate the clusters** with local Moran statistics (LISA). It then repeats the tests with another definition of neighbour, and with a dataset that does not cover every municipality. ```{python} #| eval: false !pip install -q geo4bolivia ``` ## The data The running example is the **multidimensional poverty index** (MPI) of the SDG Atlas 2026, `sdg1_2_mpi` in the dataset `sdg01`. It multiplies the share of households deprived in three or more of nine indicators (water, electricity, sanitation, literacy, identity card, telephone, attended births, teenage pregnancy, school attendance) by the average share of the nine that those households lack. It runs from 0 to 1, and **lower is better**. The Atlas computes it from the censuses of 2012 and 2024 for the 343 municipalities. ```{python} import geo4bolivia as geo import pandas as pd mpi = geo.load("sdg01", variables="sdg1_2_mpi", groups=["department"]) cols = ["variable", "year", "count", "mean", "median", "min", "max"] geo.describe(mpi)[cols] ``` Poverty fell a great deal: the mean of the index went from 0.234 to 0.063, and its highest value from 0.563 to 0.296. In 2024 the distribution is skewed (a few values far above the rest): the mean is above the median and the maximum is more than five times the median. That skew matters for the first step. ## 1. A map in classes Without further arguments `geo.map()` colours each unit on a continuous scale. With `scheme=` it sorts the units into `k` classes, each with one colour and its range in the legend. Classes are easier to read than a gradient, but someone has to decide where one class ends and the next begins, and the three schemes decide differently: | `scheme=` | How it sets the limits | Good for | Risk | | --- | --- | --- | --- | | `"quantiles"` | the same number of units in each class | comparing ranks; skewed variables | puts very different values in one class, and similar values in different classes | | `"equal_interval"` | classes of the same width | values with a meaningful, even scale | on a skewed variable most units fall in one class | | `"fisher_jenks"` | natural breaks: limits where the values have gaps, so each class holds similar values | showing the groups that are in the data | limits that are hard to explain and change with the data | The next chunk classifies the 2024 values in five classes with each scheme and counts the municipalities per class. A classed map keeps its limits in `fig.layout.meta["bins"]`. ```{python} mpi24 = mpi.loc[mpi["year"] == 2024, "sdg1_2_mpi"] rows = [] for scheme in ["quantiles", "equal_interval", "fisher_jenks"]: fig = geo.map(mpi, "sdg1_2_mpi", year=2024, scheme=scheme, k=5) limits = fig.layout.meta["bins"] units = pd.cut(mpi24, limits, include_lowest=True).value_counts(sort=False) rows.append({"scheme": scheme, "limits": [round(b, 3) for b in limits], "units per class": units.tolist()}) pd.DataFrame(rows) ``` The same 343 values give three different pictures. Quantiles put about 69 municipalities in each class (ties make the counts uneven), so the top class runs from 0.092 all the way to 0.296. Equal intervals cut the range into five steps of 0.059: 203 municipalities fall in the lowest class and one in the highest. Natural breaks sit in between, with 107, 119, 72, 36 and 9. ```{python} geo.map(mpi, "sdg1_2_mpi", year=2024, scheme="quantiles", k=5) ``` ```{python} geo.map(mpi, "sdg1_2_mpi", year=2024, scheme="equal_interval", k=5) ``` On the quantile map a fifth of the country is in the darkest colour by construction; on the equal-interval map almost all of it is pale and only the few highest values stand out. Neither is wrong. The first answers "which municipalities are in the highest fifth?", the second "which municipalities are far above the others?". Decide which question the map should answer, say which scheme it uses, and do not read a pattern off one scheme alone. ### Limits of your own, and two years in the same classes `bins=` sets the limits by hand, which is the right choice when the values have thresholds with a meaning. The Atlas counts the poverty target as met at an index of 0.066 and sees major challenges from 0.23 on. With a list of years, `geo.map()` draws one map per year **in the same classes**, so a colour means the same thing in both. ```{python} geo.map(mpi, "sdg1_2_mpi", year=[2012, 2024], bins=[0.066, 0.23]) ``` ```{python} cut = pd.cut(mpi["sdg1_2_mpi"], [0, 0.066, 0.23, 1], labels=["up to 0.066", "0.066 to 0.23", "above 0.23"]) pd.crosstab(mpi["year"], cut) ``` In 2012, 12 municipalities met the target and 163 were above 0.23. In 2024, 221 meet it and one is above 0.23. Because the classes are fixed, the two maps and the two rows of the table can be compared directly. With a `scheme` instead of `bins`, the classes of a list of years are computed over all the years shown together. Maps drawn one year at a time would each get their own classes, and the same colour would stand for different values. ### The map of the change `change=True` maps the last year minus the first, on a diverging scale centred at zero; with a scheme, falls and rises get their own classes, split at zero. ```{python} geo.map(mpi, "sdg1_2_mpi", change=True, scheme="quantiles", k=4) ``` ```{python} chg = geo.change(mpi, "sdg1_2_mpi") print(round(chg["sdg1_2_mpi_change"].corr(chg["sdg1_2_mpi_2012"]), 2)) chg["sdg1_2_mpi_change"].describe().round(3) ``` Here there is only one side: the index fell in every municipality, by between 0.015 and 0.402, so all four classes are falls. The map shows where poverty fell most in index points, which is mostly where it was highest in 2012 (the correlation between the change and the 2012 level is −0.95); it is not a map of who is poor now. ## 2. Who is a neighbour? Spatial weights Every statistic below compares a unit with its neighbours, so "neighbour" needs a definition. **Spatial weights** are that definition: for each unit, the list of its neighbours and the weight of each. `geo.weights()` builds them from the boundaries. | `method=` | Neighbours of a unit | Note | | --- | --- | --- | | `"queen"` (default) | the units that share a border or a corner with it | the number of neighbours varies from unit to unit | | `"rook"` | the units that share a border (not just a corner) | nearly the same as queen on irregular polygons | | `"knn"` | the `k` units with the nearest centroids | every unit has exactly `k`; neighbours need not touch, and A can be a neighbour of B without B being one of A | | `"distance_band"` | the units with centroids within `threshold` metres | many neighbours where units are small, few where they are large | | `"inverse_distance"` | the units within the distance band, weighted by distance ** −`power` | nearer units count more | By default the weights are **row-standardized**: each unit's weights sum to one, so "the neighbours' value" is a plain average of the neighbours. ```{python} w_queen = geo.weights("municipality") w_rook = geo.weights("municipality", method="rook") w_knn = geo.weights("municipality", method="knn", k=6) w_band = geo.weights("municipality", method="distance_band") rows = [] for name, w in [("queen", w_queen), ("rook", w_rook), ("6 nearest", w_knn), ("distance band", w_band)]: rows.append({"weights": name, "units": w.n, "mean neighbours": round(w.mean_neighbors, 2), "fewest": w.min_neighbors, "most": w.max_neighbors}) pd.DataFrame(rows) ``` Under queen contiguity a municipality has 5.5 neighbours on average, between 1 and 12. Rook is almost the same (5.4). The distance band is very different: its default threshold is the smallest distance at which every unit has at least one neighbour, about 152 km here, which is set by the most remote municipality and gives the others 43 neighbours on average and up to 87. A distance band suits units of similar size; Bolivia's municipalities are not. ### The island Contiguity leaves a unit that touches no other without neighbours: an **island**. Its "neighbours' average" is undefined, so by default (`attach_islands=True`) the package links it to its nearest unit. There is one on this map: ```{python} alone = geo.weights("municipality", attach_islands=False).islands names = geo.units("municipality").set_index("geo_id")["name"] nearest = w_queen.neighbors[alone[0]] print(alone, names[alone].tolist(), "->", names[nearest].tolist()) ``` Coipasa, in Oruro, shares no border with another municipality on this map, which leaves the salt flat of Coipasa outside every municipality. It is attached to Uru Chipaya, the nearest unit. One link among almost a thousand does not move a national statistic, but it decides everything about that unit's local statistic, so it is worth knowing it is there. ### Look at the graph `geo.connectivity()` draws the weights as a graph, one line per pair of neighbours, and counts the neighbours of each unit. Look at it before trusting any result that uses the weights: missing links, an unexpected island, or a region cut off from the rest show at once. ```{python} net = geo.connectivity("municipality") net.fig ``` ```{python} print(net.n_components, "component |", round(net.pct_nonzero, 2), "% of pairs") net.df.sort_values("n_neighbors").head(5) ``` The graph is in one piece (one *component*: every municipality can be reached from every other through a chain of neighbours), and 1.6% of all pairs of municipalities are neighbours. Five municipalities have a single neighbour. Three of them (Copacabana, Santiago de Huata and Catacora) are in La Paz and one (Bermejo) is in Tarija; the fifth is Coipasa. For these units "the neighbours' average" is the value of one other municipality, so their local statistics below rest on very little. At the other end, Villa Tunari and Charagua have 12 neighbours each. `net.fig_hist` shows the whole distribution. ## 3. Global Moran's I: is the map clustered? **Moran's I** answers one question for the whole map: do neighbouring units have more similar values than they would if the same values were dealt out at random? It is the correlation between each unit's value and the average of its neighbours: - near **zero** (slightly below: −1/(n−1)) when there is no spatial pattern, - **positive** when similar values are next to each other (clustering), - **negative** when high values sit next to low ones (a checkerboard). ```{python} res = geo.moran(mpi, "sdg1_2_mpi", year=2024) res ``` **The scatterplot.** Each point is a municipality. The horizontal axis is its value, standardized (in standard deviations from the national mean); the vertical axis is the average standardized value of its neighbours. The slope of the line is Moran's I: 0.415 in 2024. The four quadrants are the four kinds of unit: ```{python} res.df["quadrant"].value_counts() ``` 258 of the 343 municipalities are in the two quadrants where a unit resembles its neighbours: 154 below the mean among neighbours below the mean (`LL`), and 104 above the mean among neighbours above it (`HH`). The other 85 differ from their surroundings. **The test.** Is 0.415 more than chance would give? The answer comes from **permutations**: the 343 values are shuffled over the map at random, 999 times, and Moran's I is computed each time. If the map had no spatial pattern, the observed value would look like one of the shuffled ones. ```{python} print(round(res.moran_i, 3), res.p_sim, round(res.z_sim, 1), "|", res.w_spec) ``` It does not: none of the 999 shuffles reached 0.415. The pseudo p-value `p_sim` is (shuffles at least as large + 1) / (999 + 1) = 0.001, the smallest value 999 permutations can give, and the observed I is 11.9 standard deviations above the average of the shuffles (`z_sim`). `seed=` fixes the shuffles so the result is the same on every run, and `permutations=` sets how many there are. **One year, or all of them.** `year=` picks the year (the last one by default), and `year="all"` computes Moran's I in every year, on the units that have data in all of them. ```{python} geo.moran(mpi, "sdg1_2_mpi", year="all").df ``` Moran's I is 0.436 in 2012 and 0.415 in 2024. Poverty fell everywhere, yet where it is high *relative to the national mean of the year* is about as clustered as before. The amount of the fall is clustered too. A table without a `year` column, such as the one `geo.change()` returns, works the same way: ```{python} round(geo.moran(chg, "sdg1_2_mpi_change").moran_i, 3) ``` **What Moran's I does not show.** It is one number for the country. It does not say where the clusters are, how many there are, or whether they are of high or of low values. It does not say why neighbours are alike: shared geography, shared history, people and services crossing borders, or a boundary drawn through one community would all give a positive value. And it is always relative to the weights used. ## 4. LISA: where are the clusters? **LISA** (local indicators of spatial association; here the *local* Moran statistic) repeats the comparison unit by unit. Each municipality gets its own statistic and its own permutation test: its value stays fixed while the values of the others are shuffled, and the test asks whether its neighbours are more like it (or more unlike it) than randomly drawn units would be. Units that pass the test at the level `alpha` are assigned to one of four types by their quadrant in the scatterplot; the rest are "Not significant". | Type | The unit | Its neighbours | Called | | --- | --- | --- | --- | | High-High | above the mean | above the mean | a hot spot | | Low-Low | below the mean | below the mean | a cold spot | | High-Low | above the mean | below the mean | a spatial outlier | | Low-High | below the mean | above the mean | a spatial outlier | For a poverty index, "high" is bad: a High-High cluster is a group of municipalities with high poverty, and a Low-Low cluster one with low poverty. ```{python} #| classes: g4-wide lisa = geo.lisa(mpi, "sdg1_2_mpi", year=2024) lisa ``` At the 5% level, 94 of the 343 municipalities are assigned to a type: 35 High-High, 51 Low-Low and 8 outliers. The other 249 are not significant. `.df` has one row per unit with its local statistic, p-value and type, so the types can be turned into a table: ```{python} where = lisa.df.merge(mpi.loc[mpi["year"] == 2024, ["geo_id", "department"]]) where.groupby(["department", "cluster"]).size().unstack(fill_value=0) ``` Most Low-Low municipalities (low poverty among low poverty) are in Santa Cruz: 29 of the 51, more than half of that department's 56 municipalities. The High-High ones are spread over Potosí (11), Chuquisaca (9), Cochabamba (7) and Oruro (5). La Paz, with 87 municipalities, has 78 that are not significant. The figure shows each type twice: on the Moran scatterplot (left), where it is a quadrant, and on the map (right); a click on a type in the legend shows or hides it in both. `panels="map"` or `panels="scatter"` gives one of them alone, and `.fig_map` and `.fig_scatter` hold each one. ### Significance and `alpha` The types depend on the threshold. `alpha=` changes it: ```{python} counts = {} for alpha in [0.10, 0.05, 0.01]: res_a = geo.lisa(mpi, "sdg1_2_mpi", year=2024, alpha=alpha) counts[f"alpha = {alpha}"] = res_a.df["cluster"].value_counts() pd.DataFrame(counts).fillna(0).astype(int) ``` From 131 municipalities in a cluster or outlier at the 10% level to 94 at 5% and 49 at 1%. There is no correct value. The reason for caution is that the map is the result of 343 tests, one per unit, with no adjustment for making so many: with no spatial pattern at all, a 5% level would still flag about 17 municipalities (5% of 343). Here 94 are flagged, far more than that, but any single municipality near the threshold could be one of the 17. The core of a cluster, which stays at 1%, is more reliable than its edge. ### What a cluster map shows, and what it does not - **A coloured unit is the centre of a cluster, not its extent.** A High-High municipality is one whose *neighbours* are high. The neighbours belong to the cluster too, but each is coloured only if its own neighbourhood passes the test. - **High and low are relative to the mean of the year.** A Low-Low municipality of 2012 could have more poverty than a High-High one of 2024. A cluster map of each year shows where the high and low values are, not whether poverty rose or fell. To compare two years, see [Transitions and mobility](transitions.qmd). - **"Not significant" is not "average".** It means the test could not tell the unit's surroundings from a random draw. A municipality with one or two neighbours has little chance of passing whatever its value. - **It does not explain.** A cluster is a place to look at, not a finding about causes. ## 5. Do the conclusions depend on the weights? They can, so check. Here is Moran's I of 2024 under six definitions of neighbour. A panel given to `geo.weights()` builds the weights for the units that have data. ```{python} specs = {"queen": geo.weights(mpi), "rook": geo.weights(mpi, method="rook"), "4 nearest": geo.weights(mpi, method="knn", k=4), "6 nearest": geo.weights(mpi, method="knn", k=6), "10 nearest": geo.weights(mpi, method="knn", k=10), "distance band": geo.weights(mpi, method="distance_band")} rows = [] for name, w in specs.items(): m = geo.moran(mpi, "sdg1_2_mpi", year=2024, w=w) rows.append({"weights": name, "moran_i": round(m.moran_i, 3), "p_sim": m.p_sim, "z_sim": round(m.z_sim, 1)}) pd.DataFrame(rows) ``` The conclusion holds: Moran's I is positive and no shuffle reaches it under any of the six. Its size does change, from 0.43 with the four nearest neighbours to 0.33 with ten and 0.14 with the distance band. That is expected: the more units count as neighbours, and the farther away they are, the less the "neighbours' average" resembles the unit itself. So a Moran's I is comparable with another only when both use the same weights, and a report of it should always say which. The local results are more sensitive than the global one. The next table crosses the type of each municipality under queen contiguity (rows) with its type under the six nearest neighbours (columns). ```{python} knn = geo.lisa(mpi, "sdg1_2_mpi", year=2024, w=specs["6 nearest"]) both = lisa.df[["geo_id", "cluster"]].merge( knn.df[["geo_id", "cluster"]], on="geo_id", suffixes=("_queen", "_knn")) print(round((both["cluster_queen"] == both["cluster_knn"]).mean(), 2)) pd.crosstab(both["cluster_queen"], both["cluster_knn"]) ``` 83% of the municipalities (286 of 343) get the same type. No municipality changes from one type to a different one: every disagreement is between a type and "Not significant". 12 of the 35 High-High municipalities of the queen weights are not significant with the six nearest neighbours, and 25 municipalities that were not significant become Low-Low. The centres of the clusters are stable and their edges are not, which is the same lesson as with `alpha`. ## 6. A dataset with fewer units Not every dataset covers the 343 municipalities. Night-time lights per capita (`ntl`, in logarithms as `ln_NTLpc`, 2012–2020) are published for 333: the values were computed on the municipal map of 2015, so the six municipalities that lost territory since and the four units created since have no rows (`geo.info("ntl")` explains it). Some municipalities also lack a value in some years. Nothing changes in the calls. The functions use the units that have data, and build the weights among those units only: ```{python} ntl = geo.moran("ntl", "ln_NTLpc", year=2020) print(round(ntl.moran_i, 3), ntl.p_sim, "|", ntl.w_spec) ``` The description of the weights ends in `n=328`: the 328 municipalities with a value in 2020. With `year="all"`, Moran's I is computed on the municipalities that have a value in every year, so that the years are comparable: ```{python} geo.moran("ntl", "ln_NTLpc", year="all") ``` That leaves 316 municipalities (`n_obs`). Moran's I of night-time lights per capita stays between 0.42 and 0.46 in the nine years: lights are clustered, and about as much in 2020 as in 2012. Two things to keep in mind with partial coverage. A unit without data is not a neighbour of anyone: the municipalities around it have one neighbour fewer, and one that loses all of them is attached to its nearest unit, like the island above. And the result describes the units in the analysis, not the country: here 328 or 316 municipalities out of 343. Always check `n` in `w_spec` or `n_obs` before comparing results between datasets. ## In short - A classed map depends on its classes. Quantiles, equal intervals and natural breaks give different pictures of the same values; use `bins=` when thresholds have a meaning, and common classes to compare years. - Spatial weights define "neighbour". Queen contiguity is the default; look at the graph with `geo.connectivity()` and know its islands and its least connected units. - Global Moran's I tests the whole map against random placement. Multidimensional poverty is clustered in both censuses (0.44 and 0.42 under queen contiguity), although it fell in every municipality. - LISA locates the clusters: 35 High-High and 51 Low-Low municipalities in 2024 at the 5% level. Their centres are stable when `alpha` or the weights change; their edges are not. - The sign and significance of the global result hold under every definition of neighbour tried here; its size does not, so always report the weights. - With partial coverage the functions use the units with data. Check how many that is. - All of this describes a pattern. None of it says what produced it. ## Where next - **[Transitions and mobility](transitions.qmd)**: did the clusters persist between the two periods? - **[Regional inequality](inequality.qmd)**: how unequal the municipalities are, and the spatial Gini. - **[Regression, panel and spatial models](models.qmd)**: what to do when the residuals of a regression are clustered. - **[Spatial heterogeneity: MGWR](heterogeneity.qmd)**: relationships that change from place to place. - **[Analyze](../analyze.qmd)**: all the families on one question. - Reference: [`map`](../reference/map.qmd), [`weights`](../reference/weights.qmd), [`connectivity`](../reference/connectivity.qmd), [`moran`](../reference/moran.qmd), [`lisa`](../reference/lisa.qmd). ==================================================================================================== # Transitions and mobility (https://quarcs-lab.github.io/geo4bolivia/articles/transitions.html) ==================================================================================================== --- title: "Transitions and mobility" description: "How units move inside a distribution: changes of rank between two years, moves between spatial cluster types, and Markov transition matrices with and without the neighbours." image: ../images/thumbs/transitions.webp image-alt: "Cable-car cabins lit rose and turquoise gliding over La Paz at dusk, with Illimani behind" --- ```{=html}

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab
``` A mean, a Gini index or a convergence slope describe a distribution as a whole. They do not say **who moved**. Two years can have the same dispersion with every unit in the same place, or with the top and the bottom having swapped. This article is about the movement of units inside the distribution, with four functions: | Question | Function | Years needed | | --- | --- | --- | | How much did the ranking change, and who climbed or fell? | `geo.rank_mobility` | 2 | | Did the spatial clusters stay where they were? | `geo.lisa_transitions` | 2 | | How likely is a unit to move from one class of the distribution to another in a year? | `geo.transitions` | 2, better many | | Do those chances depend on the unit's neighbours? | `geo.spatial_transitions` | 2, better many | The first two compare two years, and run here on the two periods of the Sustainable Development Index. The last two follow units year by year, and run on nine years of night-time lights. They use the giddy library, which the `[dynamics]` extra installs. ```{python} #| eval: false !pip install -q "geo4bolivia[dynamics]" ``` ```{python} import pandas as pd import geo4bolivia as geo ids = geo.load("sdg", variables="index_sdg", groups=["department"]) ``` ## Rank mobility: who climbed and who fell? `geo.rank_mobility()` ranks the units in each of two years (rank 1 is the highest value) and compares the two rankings. ```{python} rm = geo.rank_mobility(ids, "index_sdg") rm ``` Each point is a municipality: its rank in 2012 against its rank in 2024. Points on the dashed line kept their position; the further from the line, the larger the move. Two **rank correlations** summarise how much of the order survived: ```{python} print("Spearman:", round(rm.spearman, 3), " Kendall:", round(rm.kendall, 3)) print("positions moved, mean:", round(rm.mean_abs_change, 1), " median:", rm.median_abs_change) print(rm.n_climbed, "climbed,", rm.n_fell, "fell, of", rm.n_units) ``` - **Spearman's correlation** is the ordinary correlation computed on the ranks: 1 if the order is identical, 0 if the two rankings are unrelated. Here it is 0.84. - **Kendall's tau** counts pairs of units. It is the share of pairs in the same order in both years minus the share that swapped: 0.67, which means about 83% of all pairs of municipalities kept their order. - **A municipality moved 40 positions on average** (the median is 29) in a ranking of 343. Three kept their rank: Nuestra Señora de La Paz, Tarija and Santa Cruz de la Sierra, at positions 1, 2 and 6. So the order was broadly kept, with a good deal of reshuffling. `rm.gt` lists the units that moved most (`top=` sets how many), and `rm.climbers` and `rm.fallers` hold them as tables: ```{python} rm.gt ``` The largest climb is Cruz de Machacamarca, from rank 315 to 90, with an index that rose from 42.9 to 63.4. The largest fall is San José de Chiquitos, from 42 to 283: it is the one municipality whose index went down (57.8 to 55.1). The other units in the lower half of the table all improved their index, by less than the rest. A fall in rank is not a fall in the value. ### The quantile transition matrix The second view sorts the units into `k` groups of equal size in each year (quintiles by default; Q1 has the lowest values) and counts where the members of each group ended up. ```{python} rm.fig_matrix ``` Each row is a group of 2012 and sums to one. The diagonal is the share that stayed: 56% of all municipalities are in the same quintile in both years (`rm.share_same_group`), against the 20% expected if the two years were unrelated. The ends hold their members best: 77% of the top quintile and 67% of the bottom one stayed, against 37% of the middle quintile. That is partly arithmetic: a unit at the top can only move down, and in the middle of a distribution values are close together, so a small change in the index crosses many positions. What rank mobility does not show: - **Levels.** Ranks are relative. The index rose in 342 of the 343 municipalities, and half of them still fell in the ranking, because a ranking has as many moves down as up. - **Distances.** A move of 50 positions in the crowded middle can be a smaller change of the index than a move of 5 positions at the top. - **Noise.** With two periods, a unit whose first value was unusually low or high will move towards its usual place, and that looks like mobility. For variables where less is better (poverty, mortality), `ascending=True` makes rank 1 the lowest value. ## LISA transitions: did the clusters stay in place? Local Moran statistics (LISA) give each unit a type according to its value and its neighbours' values, both compared with the mean: **High-High** (a high value among high neighbours, a hot spot), **Low-Low** (a cold spot), **High-Low** and **Low-High** (outliers that differ from their surroundings), or **Not significant** when the pattern is compatible with chance. The [spatial dependence article](spatial-dependence.qmd) explains them. `geo.lisa_transitions()` computes the types in two years, on the same units and the same weights, and counts the moves between them. ```{python} lt = geo.lisa_transitions(ids, "index_sdg") lt ``` Rows are the types in 2012, columns the types in 2024, and each cell counts municipalities. - **Most units kept their type**: 275 of 343 (80%). Most of those, 204, are not significant in either year. - **Hot spots persisted and grew in number.** Of the 46 High-High municipalities of 2012, 39 are still High-High in 2024, and 17 that were not significant joined them: 60 in 2024. - **Cold spots persisted less.** Of the 44 Low-Low municipalities of 2012, 30 are still Low-Low; 13 are no longer significant, and 13 others became Low-Low: 45 in 2024. - **Nothing crossed over.** No municipality went from High-High to Low-Low or the other way. Global Moran's I, the overall tendency of similar values to be neighbours, changed little: 0.43 in 2012 and 0.47 in 2024 (`lt.moran_i_first`, `lt.moran_i_last`). The moves in words, from the most common: ```{python} lt.df[lt.df["moved"]]["transition"].value_counts().head(6) ``` The map shows the types of 2024, with the municipalities that changed type outlined: ```{python} lt.fig_map ``` Read the changes with care: - **The threshold.** A unit has a type only when its pseudo p-value is below 0.05 (`alpha=`). Units near the threshold move in and out of "Not significant" without much having changed, and 58 of the 68 moves here are to or from that category. - **Relative to each year's mean.** "High" and "low" are measured against the mean of the same year. The index rose almost everywhere, so a Low-Low municipality of 2024 usually has a higher index than it had in 2012. - **The neighbours count.** A unit can change type because its neighbours' values changed while its own did not. ## Markov transitions: how likely is a move? With many years, the same counting can be done for every pair of consecutive years and turned into probabilities. `geo.transitions()` assigns each unit to one of `k` classes in every year and builds the **transition matrix**: for a unit in a given class this year, the probability of being in each class next year. The data are night-time lights per capita (`ntl`, the variable `ln_NTLpc`), yearly from 2012 to 2020. ```{python} tr = geo.transitions("ntl", "ln_NTLpc", k=5) tr ``` ```{python} print(tr.notes) tr.p.round(3) ``` The classes are quintiles formed in each year, so they are positions in that year's distribution: Q1 is the fifth of municipalities with the least light per capita. (The variable is a logarithm; quintiles do not change when a variable is logged, so that makes no difference here.) The matrix pools 2,528 yearly moves of the 316 municipalities observed in all nine years. - **Positions are persistent from one year to the next.** A municipality stays in its class with a probability of 0.83 on average. The ends are the most persistent: 0.90 for Q1 and 0.91 for Q5; in the middle classes it is 0.76 to 0.82. - **Moves are short.** Almost all the probability off the diagonal is in the classes next to it. From Q1, the chance of reaching Q3 in one year is 0.002, and no unit went further. The table of the result adds three kinds of summary: ```{python} tr.gt ``` - **Mobility indices** condense the matrix in one number. Shorrocks' index is 0 when no unit ever moves and 1 when next year's class does not depend on this year's: 0.21 here. - **Expected stay** (sojourn time) is the number of consecutive years a unit is expected to remain in a class, 1 / (1 − the probability of staying): about 10 years in Q1, 11 in Q5 and 4 in Q2. - **The steady state** is the distribution the units would settle into if the same matrix applied year after year, for ever. Here the steady state is 20% in each class, and it could be nothing else: quintiles formed year by year hold 20% of the units every year by construction. To ask where the distribution is heading, the classes have to be fixed levels. `per_year=False` forms them over all the years together, and `relative=True` first divides each value by the mean of its year, so that a class is a position relative to the average municipality: ```{python} fixed = geo.transitions("ntl", "ln_NTLpc", k=5, per_year=False, relative=True) fixed.steady_state.round(2) ``` With fixed classes the long-run distribution has fewer units at both ends (17% in the lowest class and 12% in the highest, against 20% of the observations in each) and more in the middle. This is the same movement that [σ-convergence](convergence.qmd) measures on these data as a falling dispersion, seen through the transition matrix. The steady state is an extrapolation. It assumes that the probabilities are the same every year (eight yearly steps are little to check that), that a unit's chances depend only on its present class and not on how it got there, and that the same matrix will hold in the future. Read it as "the direction in which these eight years of moves point", not as a forecast. ## Spatial Markov: do the neighbours matter? The transition matrix treats all units in a class alike. Rey's **spatial Markov** chain asks whether a unit's chances depend on where its neighbours stand. `geo.spatial_transitions()` classifies each unit's neighbourhood (the average of its neighbours, the *spatial lag*) as well as the unit, and estimates one transition matrix for each class of neighbourhood. If space does not matter, the matrices are equal apart from chance, and two tests check that. By default the classes are fixed levels relative to each year's mean. Three classes instead of five keep enough moves in every row of every matrix: ```{python} st = geo.spatial_transitions("ntl", "ln_NTLpc", k=3) st ``` ```{python} st.gt ``` Each panel of the figure is the transition matrix of the units whose neighbours are in one class, from the lowest (Q1) to the highest (Q3). Compare the same row across panels: ```{python} pd.concat({f"neighbours in Q{i + 1}": p.round(3) for i, p in enumerate(st.p_conditional)}) ``` - **A municipality in the lowest class is more likely to leave it when its neighbours are higher.** The probability of moving from Q1 to Q2 in a year is 0.06 with neighbours in the lowest class, 0.11 with neighbours in the middle class and 0.17 with neighbours in the highest. - **A municipality in the highest class is more likely to keep it when its neighbours are high too.** The probability of falling from Q3 to Q2 is 0.10, 0.08 and 0.04 across the same three neighbourhoods. - **The tests reject equal matrices**: the likelihood ratio test gives p = 0.010 and the chi-square test p = 0.008, with 10 degrees of freedom. The average probability of staying is about the same in low and in high neighbourhoods, which is what the reading above reports; the difference is in the *direction* of the moves. Low neighbours go with staying low, high neighbours with staying high or moving up. What this does not show: - **Not a spillover.** Neighbouring municipalities can share roads, climate, markets and administrative history. Any of these makes neighbours move together without one affecting the other. - **Thin cells.** The moves are split over classes of units and classes of neighbourhood, and some combinations are rare: municipalities in the lowest class with neighbours in the highest account for 60 of the 2,528 moves. With `k=5` the same row rests on 7. Check the counts before reading a probability: ```{python} moves = st.df.sort_values(["geo_id", "year"]) moves["next_state"] = moves.groupby("geo_id")["state"].shift(-1) pd.crosstab([moves["neighbor_state"], moves["state"]], moves["next_state"]) ``` - **The units left out.** Only municipalities with lights in every year enter (316 of 333), and a unit's neighbours are those among the 316. ## A transition matrix from two periods `geo.transitions()` also runs on two years. On the Sustainable Development Index it gives the matrix that `rank_mobility` showed above: ```{python} two = geo.transitions(ids, "index_sdg", k=5) print(two.n_transitions, "moves, one per municipality") two.p.round(2) ``` What it **can** say: how the municipalities of each quintile of 2012 are distributed over the quintiles of 2024. That is a description of one observed step, and a useful one. What it **cannot** say: - **Whether the probabilities are stable.** There is one step, so there is nothing to compare it with. Each row rests on about 69 municipalities. - **Anything per year.** The step is about twelve years long. An "expected stay of 4 periods" in this matrix would mean 4 steps of twelve years. - **Where the distribution is heading.** A steady state repeats the same step for ever. With one observed step, between two nominal years, that is arithmetic and not evidence. - **Real moves from noise.** A unit near the border of two classes changes class with a small change of its value, or with an error in either year. With many years such flips show up as moves back and forth; with two they cannot be told apart from lasting moves. With two periods, stay with the description: `rank_mobility` for who moved and by how much, and `lisa_transitions` for the clusters. ## In short - **`geo.rank_mobility`**: between ~2012 and ~2024 the ranking of the Sustainable Development Index was broadly kept (Spearman 0.84, Kendall 0.67), a municipality moved 40 positions on average, and 56% stayed in their quintile. - **`geo.lisa_transitions`**: 80% of the municipalities kept their LISA type; hot spots went from 46 to 60 and cold spots from 44 to 45, and most changes are to or from "Not significant". - **`geo.transitions`**: from one year to the next, a municipality stays in its quintile of night-time lights per capita with a probability of 0.83, and moves are almost always to the next class. - **`geo.spatial_transitions`**: those chances differ with the neighbours' class (p = 0.01): low neighbours go with staying low, high neighbours with moving up. - Classes formed year by year describe positions; a steady state needs fixed classes, and is an extrapolation in any case. - A transition matrix from two periods describes one step. It does not support long-run statements. ## Where next - **[Convergence: β, σ and clubs](convergence.qmd)**: whether the distribution narrows, and whether units head for one path. - **[Maps and spatial dependence](spatial-dependence.qmd)**: Moran's I, LISA and the spatial weights behind this article. - **[Regional inequality](inequality.qmd)**: how unequal the distribution is in each year. - **[Exploring the data](exploring.qmd)**: distributions, extremes and trends before any of this. - **[Case study: the Sustainable Development Index](case-study-sdg.qmd)**: mobility as one step of a complete analysis. ==================================================================================================== # Function reference ==================================================================================================== ## geo.datasets(ref: 'str | None' = None) -> 'pd.DataFrame' The catalog: one row per dataset and scale. Returns ------- pandas.DataFrame Columns ``dataset`` (the id to pass to ``geo.load``), ``scale``, ``title``, ``years`` (e.g. ``"2001–2020"`` for an annual panel or ``"2012, 2024"`` for two periods), ``units`` (units with data), ``variables`` (how many), ``first_year``, ``last_year``, ``license``, ``categories`` and the ``csv`` and ``parquet`` paths. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px cat = geo.datasets() display(cat[["dataset", "scale", "title", "years", "units", "variables"]]) px.bar(cat, x="dataset", y="units", color="years", hover_name="title", title="Units with data per dataset") ``` ## geo.search(text: 'str', dataset: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame' Find variables whose name, label or description contain every word of ``text``. Matching ignores case and accents (``"educacion"`` finds "Educación"). Results are ranked: matches in the variable name or label come first. Parameters ---------- text One or more words, e.g. ``"poverty"`` or ``"night lights"`` dataset Search only this dataset Returns ------- pandas.DataFrame One row per variable and scale: ``dataset``, ``variable``, ``label``, ``unit``, ``years``, ``scale``, ``units`` (with data) and ``description``. Pass the ``dataset`` and ``variable`` to ``geo.load(dataset, variables=variable)``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px hits = geo.search("education women") # every word must match display(hits[["dataset", "variable", "label", "years"]]) df = geo.load(hits["dataset"][0], variables=hits["variable"][0], names=True) px.box(df, x="year", y=hits["variable"][0], hover_name="name", labels=geo.labels(df), title=hits["label"][0]) ``` ## geo.info(dataset: 'str', ref: 'str | None' = None) -> 'DatasetInfo' What a dataset is: description, coverage (scales, units, years), actual reference years, license, sources with their licenses, citation, notes and notebooks. Parameters ---------- dataset Dataset id (see ``geo.datasets()``) Returns ------- DatasetInfo A ``DatasetInfo``: in a notebook it displays as formatted HTML; ``print()`` gives plain text, and ``.dict`` the full datapackage. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px geo.info("sdg06") # description, coverage, sources, license, notes ``` ## geo.dictionary(dataset: 'str', scale: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame' The variable dictionary of a dataset, one row per column. Parameters ---------- dataset Dataset id (see ``geo.datasets()``) scale The resource's scale (default: the dataset's native scale) Returns ------- pandas.DataFrame Columns ``variable``, ``label`` (the field's ``title``, a readable name for plot axes and legends), ``description`` (how it was built), ``unit``, ``role`` (id, time, source_key or variable), ``aggregation`` (the rule ``geo.aggregate`` uses), ``source`` and ``reference_years`` (the actual years behind each stored year, e.g. ``"2012: 2008; 2024: 2023"``, when they differ). Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px d = geo.dictionary("sdg03") display(d[["variable", "label", "unit", "aggregation", "reference_years"]]) ``` ## geo.reference_years(dataset: 'str', variable: 'str | None' = None, scale: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame' The actual years behind each stored year, one row per variable and stored year. Some panels store values at nominal years: the SDG Atlas, for example, stores its initial values at 2012 and its final values at 2024, while an indicator may come from a survey of 2008 and 2023. Parameters ---------- dataset Dataset id variable One variable (default: all) scale The resource's scale (default: the dataset's native scale) Returns ------- pandas.DataFrame Columns ``variable``, ``label``, ``year`` (as stored) and ``reference`` (the actual years; the stored year when they are the same). Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px ry = geo.reference_years("sdg03") # stored year -> actual years ry[ry["year"] == 2024] ``` ## geo.cite(dataset: 'str', style: 'str' = 'apa', ref: 'str | None' = None) -> 'str' How to cite a dataset: its recommended citation, then the credits its sources require. Parameters ---------- dataset Dataset id style ``"apa"`` (plain text) or ``"bibtex"`` Returns ------- str A string; print it: ``print(geo.cite(""))``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px print(geo.cite("sdg01")) # APA, then the sources' credits print(geo.cite("sdg01", style="bibtex")) ``` ## geo.labels(data, own: 'dict[str, str] | None' = None, ref: 'str | None' = None) -> 'dict[str, str]' Readable labels for the columns of a panel (or every field of a dataset id). Variables take their metadata ``title``; columns written by ``geo.change()`` (``_2024``, ``_change``...) are labelled from their variable; group columns (``department``) take the scale's label. Parameters ---------- data A panel (any DataFrame) or a dataset id own Labels for columns of your own, ``{column: label}``. They are stored on the table, so the figures, tables and readings of the analysis functions use them Returns ------- dict ``{column: label}`` for the columns it knows. Pass it to Plotly: ``px.scatter(df, x=..., y=..., labels=geo.labels(df))``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px df = geo.load("sdg07", variables=["sdg7_1_cae", "sdg7_1_elc"], years=2024) px.scatter(df, x="sdg7_1_cae", y="sdg7_1_elc", labels=geo.labels(df), title="Electricity and clean cooking, ~2024") ``` ## geo.describe(df: 'pd.DataFrame', variables=None, by: 'str | None' = 'year', ref: 'str | None' = None) -> 'pd.DataFrame' Summary statistics of a panel's variables, per ``by`` group (default: per year). Parameters ---------- df A panel, e.g. from ``geo.load`` variables A variable or list of them (default: every numeric variable) by A column to group by: ``"year"`` (default), ``None`` to pool all rows, or a group such as ``"department"`` (after ``geo.load(..., groups=["department"])``) Returns ------- pandas.DataFrame One row per variable (and group): ``variable``, ``label``, ``unit``, ``count``, ``mean``, ``std``, ``min``, ``p25``, ``median``, ``p75`` and ``max``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px df = geo.load("sdg06", groups=["department"]) display(geo.describe(df)) # per period s = geo.describe(df, variables="index_sdg6", by="department") px.bar(s, x="department", y="mean", error_y="std", labels={"mean": s["label"][0]}, title="Clean water and sanitation index by department (both periods)") ``` ## geo.load(dataset: 'str', scale: 'str | None' = None, years=None, variables=None, wide: 'str | None' = None, geometry: 'bool' = False, names: 'bool' = False, groups=None, ref: 'str | None' = None) Load a dataset as a panel: one row per unit (``geo_id``) and year. Parameters ---------- dataset Dataset id (see ``geo.datasets()``) scale Scale to load; defaults to the dataset's native scale years An int, a list of ints, or a ``(first, last)`` range (inclusive) variables A variable or list of variables to keep (``geo_id`` and ``year`` are always kept); ``geo.search()`` and ``geo.dictionary()`` list them wide A variable name; returns one row per unit with one column per year (````) geometry Attach unit boundaries (and names) and return a GeoDataFrame names Add the unit ``name`` groups Coarser scales whose unit names to add as columns, e.g. ``["department"]`` adds a ``department`` column (implies ``names``) ref Git branch, tag or commit to read from GitHub (default: local checkout or ``main``) Notes ----- Raises ``ValueError`` for unknown variables, and for years that have no rows (listing the years the dataset has). Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px df = geo.load("sdg05", variables=["index_sdg5"], years=[2012, 2024], groups=["department"]) px.histogram(df, x="index_sdg5", color="year", barmode="overlay", labels=geo.labels(df), title="Gender equality index") ``` ## geo.add_names(df: 'pd.DataFrame', groups=None, scale: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame' Add the unit ``name`` and, for each scale in ``groups``, the name of the unit it lies in. Parameters ---------- df Any table with ``geo_id`` groups A coarser scale or list of them (see ``geo.ancestors``), e.g. ``["department"]`` scale The units' scale (default: inferred from ``geo_id``) Returns ------- pandas.DataFrame A copy with ``name`` and one column per group (the group unit's name) after ``geo_id`` (and ``year``). Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px df = geo.load("sdg08", variables="index_sdg8", years=2024) df = geo.add_names(df, groups=["department", "province"]) px.box(df, x="department", y="index_sdg8", hover_name="name", labels=geo.labels(df), title="Decent work index, ~2024") ``` ## geo.units(scale: 'str | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame' All units (or the units of one scale) with names, parents and source keys. Parameters ---------- scale A scale id (see ``geo.scales()``); default: every scale Returns ------- pandas.DataFrame Every row has ``geo_id``, ``scale``, ``name``, ``parent_id``, ``area_km2``, a ``_id`` column for each coarser scale it lies in, and the codes each source uses for it (e.g. ``ine_code``), which ``geo.link(..., key=...)`` matches. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px m = geo.units("municipality") # names, parents, INE codes, areas display(m[["geo_id", "name", "ine_code", "province_id", "department_id"]].head()) px.histogram(m, x="area_km2", log_y=True, title="Municipal areas (km²)") ``` ## geo.boundaries(scale: 'str', ref: 'str | None' = None) -> 'gpd.GeoDataFrame' Unit polygons for a scale (CRS84), with ``geo_id`` and ``name``. Parameters ---------- scale A scale id (see ``geo.scales()``) Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px dep = geo.boundaries("department") px.choropleth_map(dep, geojson=dep.geometry, locations=dep.index, color="name", hover_name="name", center={"lat": -16.3, "lon": -64}, zoom=4, title="Bolivia's nine departments") ``` ## geo.crosswalk(source: 'str', target: 'str', ref: 'str | None' = None) -> 'pd.DataFrame' Links between two scales: ``geo_id_from``, ``geo_id_to``, ``weight_area``. Parameters ---------- source The finer scale, e.g. ``"municipality"`` target A coarser scale it nests in, e.g. ``"department"`` Returns ------- pandas.DataFrame ``weight_area`` is the share of the source unit's area inside the target unit (1.0 for nested scales). Crosswalks are published for every pair of active scales where one nests in the other, from finer to coarser. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px xw = geo.crosswalk("municipality", "department") n = xw.groupby("geo_id_to", as_index=False).size() n["department"] = n["geo_id_to"].map(geo.units("department") .set_index("geo_id")["name"]) px.bar(n, x="department", y="size", title="Municipalities per department") ``` ## geo.scales(active_only: 'bool' = False, ref: 'str | None' = None) -> 'pd.DataFrame' The registry of spatial scales as a table, one row per scale (``geo.scales()``). Columns: ``label``, ``kind`` (admin, grouping, grid, custom), ``parent`` (the coarser scale it nests in), ``children``, ``id_pattern``, ``example`` and ``status`` (active or planned). ``active_only=True`` hides planned scales. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px geo.scales() # ids, labels, parents, ID patterns ``` ## geo.ancestors(scale: 'str', ref: 'str | None' = None) -> 'list[str]' Coarser scales a unit of ``scale`` nests in, in breadth-first order. Administrative scales follow ``parent``; a scale whose units are unions of ``scale`` units (a grouping, such as statistical regions) is also an ancestor. The parent chain of each level comes before the groupings of that level, e.g. ``municipality -> [province, department, country]``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px print(geo.ancestors("municipality")) # province, department, country ``` ## geo.scale_of(geo_id: 'str', ref: 'str | None' = None) -> 'str' The scale a geo ID belongs to, from the ID patterns (e.g. a municipality ID -> ``"municipality"``). Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px print(geo.scale_of("BO-04-17-04"), geo.scale_of("BO-04")) ``` ## geo.config() -> 'dict' The project settings: the package's project.yaml, with the data repository's on top inside a checkout. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px cfg = geo.config() # project.yaml: repository, CRS... print(cfg["repository"], cfg["equal_area_crs"]) ``` ## geo.change(df: 'pd.DataFrame', variables=None, first: 'int | None' = None, last: 'int | None' = None) -> 'pd.DataFrame' The change of each unit between two years of a panel, one row per unit. Parameters ---------- df A long panel with ``geo_id``, ``year`` and variables (e.g. from ``geo.load``) variables A variable or list of them (default: every numeric variable) first, last The two years to compare (default: the panel's first and last year) Returns ------- pandas.DataFrame One row per unit. For each variable ``v``: ``v_``, ``v_``, ``v_change`` (last minus first), ``v_pct_change`` (in percent of the first value), ``v_annual_change`` (the change divided by the years between them), ``v_rank_`` and ``v_rank_`` (1 = highest value) and ``v_rank_change`` (positive when the unit climbed). Text columns that are constant within a unit (``name``, ``department``...) are kept. The years are the stored years: for panels stored at nominal years (see ``geo.reference_years``), the actual span of an indicator may differ. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px df = geo.load("sdg01", variables="sdg1_2_mpi", groups=["department"]) chg = geo.change(df, "sdg1_2_mpi") px.scatter(chg, x="sdg1_2_mpi_2012", y="sdg1_2_mpi_change", color="department", hover_name="name", labels=geo.labels(chg), title="Poverty fell most where it was highest") ``` ## geo.aggregate(df: 'pd.DataFrame', to, dataset: 'str | None' = None, rules: 'dict | None' = None, variables=None, weights: 'pd.DataFrame | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame' Aggregate a panel up to a coarser, nested scale (or to several at once). Parameters ---------- df Panel with ``geo_id``, optionally ``year``, and variable columns to Target scale id, or a list of them (e.g. ``["department", "country"]``), each containing the source scale (see ``geo.ancestors``) dataset Take each variable's ``geo:aggregation`` rule from this dataset's metadata rules Explicit rules that override the metadata, e.g. ``{"y": "weighted_mean:pop"}`` variables Variables to aggregate (default: every numeric column except ids and year, leaving out, with a warning, those without a rule) weights Extra columns to join on ``geo_id`` (and ``year`` if present) for ``weighted_mean`` rules whose weight is not already in ``df``. When the dataset's metadata names where a weight lives (``geo:weights``, e.g. census population), it is loaded automatically. Returns ------- pandas.DataFrame One row per target unit (and year) with ``n_units``, the number of source units with data for at least one of the variables, and ``coverage``, their share of all the source units in the target (1.0 when every unit has data). With a list of targets, the results are stacked with a ``scale`` column. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px df = geo.load("sdg04", variables="index_sdg4") up = geo.aggregate(df, to=["department", "country"], dataset="sdg04") up = geo.add_names(up[up["scale"] == "department"]) px.bar(up, x="name", y="index_sdg4", color="year", barmode="group", labels=geo.labels(up), title="Education index by department (weighted)") ``` ## geo.combine(datasets, scale: 'str', years=None, extra=(), rules: 'dict | None' = None, ref: 'str | None' = None) -> 'pd.DataFrame' Merge datasets into one panel at ``scale``, aggregating finer ones up. Parameters ---------- datasets Dataset ids, or ``{dataset: [variables]}`` to pick variables scale Target scale; each dataset must be published at it or nest in it years As in ``geo.load`` extra Your own panels (with ``geo_id`` and ``year``) at ``scale`` rules Aggregation rules overriding the metadata, ``{variable: rule}`` Notes ----- Variables without an aggregation rule (``none``) are dropped, with a warning, when a dataset has to be aggregated. Clashing column names get the dataset id as a prefix (``_``). Weighted means fetch their weights as ``geo.aggregate()`` does (``geo:weights`` in the metadata). Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px dep = geo.combine(["sdg03", "sdg04"], scale="department", years=2024) dep = geo.add_names(dep) px.scatter(dep, x="index_sdg3", y="index_sdg4", text="name", labels=geo.labels(dep), title="Health and education, ~2024") ``` ## geo.overlay(df: 'pd.DataFrame', target, extensive=(), intensive=(), source=None, id_col: 'str' = 'geo_id') -> 'pd.DataFrame' Area-weighted interpolation of a panel onto non-nested units. Parameters ---------- df Panel with ``geo_id``, optionally ``year``, and the variables target A scale id or a GeoDataFrame with ``id_col`` and polygons (e.g. a grid) extensive Counts/totals (population, GDP): split by the share of each source unit's area falling in the target, then summed intensive Rates/averages (GDP per capita, HDI): area-weighted mean over the parts of source units inside the target source The source scale or GeoDataFrame (default: inferred from ``geo_id``) Notes ----- Assumes values are spread uniformly within each source unit. Returns one row per target unit (and year), with ``geo_id`` holding the target ids. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px import geopandas as gpd import numpy as np from shapely.geometry import box crs = geo.config()["equal_area_crs"] # metres, equal area x0, y0, x1, y1 = geo.boundaries("country").to_crs(crs).total_bounds size = 50_000 # 50 km cells cells = [box(x, y, x + size, y + size) for x in np.arange(x0, x1, size) for y in np.arange(y0, y1, size)] grid = gpd.GeoDataFrame({"cell": range(len(cells))}, geometry=cells, crs=crs) grid = grid.to_crs("EPSG:4326") pop = geo.load("sdg", variables="population", years=2024) cells = geo.overlay(pop, grid, extensive=["population"], id_col="cell") px.histogram(cells, x="population", title="Population on a 50 km grid") ``` ## geo.overlay_weights(source, target, id_col: 'str' = 'geo_id') -> 'pd.DataFrame' Area of every source-target intersection, in an equal-area projection. Parameters ---------- source, target Scale ids, or GeoDataFrames with ``id_col`` and polygons id_col The id column of GeoDataFrame inputs Returns ------- pandas.DataFrame Columns ``geo_id_from``, ``geo_id_to``, ``area_km2``, ``share_of_from`` (the part of the source unit inside the target) and ``share_of_to``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px w = geo.overlay_weights("municipality", "department") px.histogram(w, x="share_of_from", log_y=True, title="Share of each municipality inside each department") ``` ## geo.link(df: 'pd.DataFrame', scale: 'str', on: 'str', key: 'str' = 'auto', errors: 'str' = 'raise', ref: 'str | None' = None) -> 'pd.DataFrame' Attach ``geo_id`` to a table that identifies units by name or by a source code. Parameters ---------- df Your table (one row per unit, or per unit and year) scale Scale of your units, e.g. ``"province"`` (see ``geo.scales()``) on The column of ``df`` that identifies the unit key The column of ``geo.units(scale)`` to match against: ``"name"``, a source code column of ``geo.units(scale)`` (e.g. ``"gid"`` for GADM codes), or ``"auto"``: the column matching most values, with the others filling the gaps (so mixed spellings from different sources still link). Matching ignores accents, case, spaces and punctuation ("BântéayMéanchey" = "Banteay Meanchey"). A value shared by several units (two municipalities with the same name) is ambiguous and links to none of them: link such units by a code instead. errors ``"raise"`` (default), ``"warn"`` or ``"ignore"`` for values that match no unit or are ambiguous Returns ------- pandas.DataFrame A copy of ``df`` with ``geo_id`` as its first column. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px import pandas as pd mine = pd.DataFrame({"municipio": ["SUCRE", "el alto", "Cobija"], "clinics": [41, 63, 12]}) # made-up values mine = geo.link(mine, scale="municipality", on="municipio") by_code = geo.link(pd.DataFrame({"cod": [20101]}), scale="municipality", on="cod", key="mun_id") # INE code -> geo_id geo.add_names(mine) ``` ## geo.check(df: 'pd.DataFrame', scale: 'str | None' = None, time: 'str | None' = 'year', ref: 'str | None' = None) -> 'list[str]' Problems that would stop ``df`` from being a valid panel at ``scale`` (empty list = OK). Checks: a ``geo_id`` column, ids known at the scale, one row per unit (and year) and integer years. Parameters ---------- df Your panel, e.g. from ``geo.link`` scale Its scale (default: inferred from ``geo_id``) time The year column, or ``None`` for a cross-section Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo import plotly.express as px import pandas as pd df = pd.DataFrame({"geo_id": ["BO-04-17-04", "BO-04-17-04", "BO-99"], "year": [2024, 2024, 2024], "x": [1, 2, 3]}) geo.check(df, scale="municipality") # unknown id, duplicate rows ``` ## geo.zonal(raster, scale, stats=('sum', 'mean'), name: 'str' = 'value', year: 'int | None' = None, band: 'int' = 1, all_touched: 'bool' = False, ref: 'str | None' = None) -> 'pd.DataFrame' Summarise a raster (GeoTIFF path or URL) over the units of a scale. Parameters ---------- raster Path or URL of a raster, or ``{year: path}`` for an annual series scale A scale id, or a GeoDataFrame with ``geo_id`` and polygons stats Any of ``sum``, ``mean``, ``min``, ``max``, ``median``, ``count`` name Prefix of the output columns (``name_sum``, ``name_mean``, ...) year Year to record when ``raster`` is a single file all_touched Count every pixel a unit touches (default: pixel centres inside) Notes ----- Needs ``rasterio``: in Colab, ``!pip install -q "geo4bolivia[raster]"``. Pixels are not split at unit edges, so use a fine raster, or ``all_touched=True`` for very small units. Nodata pixels are ignored. Example: ```python !pip install -q "geo4bolivia[raster]" import numpy as np import rasterio from rasterio.transform import from_bounds import geo4bolivia as geo import plotly.express as px # A small raster of rough distance to La Paz (km); use your own GeoTIFF instead x0, y0, x1, y1 = geo.boundaries("country").total_bounds lon, lat = np.meshgrid(np.linspace(x0, x1, 400), np.linspace(y1, y0, 400)) km = 111 * np.hypot(lon + 68.13, lat + 16.50) with rasterio.open("dist.tif", "w", driver="GTiff", width=400, height=400, count=1, dtype="float32", crs="EPSG:4326", transform=from_bounds(x0, y0, x1, y1, 400, 400)) as f: f.write(km.astype("float32"), 1) dist = geo.zonal("dist.tif", scale="municipality", stats=["mean", "max"], name="dist_lpz", year=2024) geo.map(dist, "dist_lpz_mean", title="Mean distance to La Paz (km)") ``` ## geo.save(df: 'pd.DataFrame', name: 'str', scale: 'str', directory='.', title: 'str | None' = None, description: 'str' = '', variables: 'dict | None' = None, sources=(), citation: 'str | None' = None, license: 'str' = 'CC-BY-4.0') -> 'dict' Save a panel exactly like the published datasets, ready to share or contribute. Parameters ---------- df A valid panel (see ``geo.check``) name The dataset id: lowercase letters, digits, ``-`` or ``_`` scale Its scale directory Where to write ``data/`` and ``metadata/`` (default: the current folder) title, description What the dataset is variables ``{column: {"label", "description", "unit", "aggregation", "source"}}`` sources Titles or ``{"title", "path"}`` dicts of the sources citation The recommended citation license An SPDX license id (default ``CC-BY-4.0``) Notes ----- Writes ``/data//.csv`` and ``.parquet`` and ``/metadata/.datapackage.json`` (Frictionless). Describe each variable in ``variables``: ``{"pop": {"label": "Population", "description": "Sum of WorldPop 100 m population over the unit", "unit": "people", "aggregation": "sum", "source": "WorldPop"}}``. The ``label`` is a readable name for plots; the description says how the variable was built. The aggregation rule (``sum``, ``mean``, ``weighted_mean:``, ``none``, ...) tells ``geo.aggregate()`` how to move the variable to coarser scales. Returns the datapackage as a dict. Example: ```python !pip install -q "geo4bolivia[raster]" import numpy as np import rasterio from rasterio.transform import from_bounds import geo4bolivia as geo import plotly.express as px # A small raster of rough distance to La Paz (km); use your own GeoTIFF instead x0, y0, x1, y1 = geo.boundaries("country").total_bounds lon, lat = np.meshgrid(np.linspace(x0, x1, 400), np.linspace(y1, y0, 400)) km = 111 * np.hypot(lon + 68.13, lat + 16.50) with rasterio.open("dist.tif", "w", driver="GTiff", width=400, height=400, count=1, dtype="float32", crs="EPSG:4326", transform=from_bounds(x0, y0, x1, y1, 400, 400)) as f: f.write(km.astype("float32"), 1) dist = geo.zonal("dist.tif", scale="province", stats=["mean"], name="dist_lpz", year=2024) pkg = geo.save(dist, "distance", scale="province", directory="my_data", title="Distance to La Paz", variables={"dist_lpz_mean": {"label": "Distance to La Paz (km)", "unit": "km", "aggregation": "none"}}) print(pkg["resources"][0]["schema"]["fields"][2]) ``` ## geo.correlation(data, variables=None, year=None, *, method: 'str' = 'pearson', digits: 'int' = 2, title: 'str | None' = None, ref: 'str | None' = None) -> 'CorrelationResult' Correlations between the variables of a panel in one year. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` variables The numeric columns to correlate (default: every numeric variable; ids and the year are left out) year The year to use (default: the last with data for the first variable), or ``"all"`` to pool every year method ``"pearson"`` (linear), ``"spearman"`` (ranks) or ``"kendall"``: the coefficient of the matrix and the heatmap digits Decimals written in the heatmap and the table title The figure and table title Notes ----- Only the observations with data for every variable are used, so that all the coefficients refer to the same units. Returns ------- CorrelationResult A result with ``.df`` (the correlation matrix), ``.pairs`` (one row per pair, from the strongest positive to the strongest negative), ``.fig`` (a heatmap centred at zero with the coefficients in the cells), ``.gt`` (a table with Pearson above the diagonal and Spearman below), ``.n`` (observations), ``.variables``, ``.year``, ``.notes`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.correlation("sdg01", year=2024) print(res.interpret()) res.fig ``` ## geo.extremes(data, variable: 'str', year=None, *, n: 'int' = 5, title: 'str | None' = None, ref: 'str | None' = None) -> 'ExtremesResult' The units with the highest and the lowest values of a variable in one year. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` variable The numeric column to rank year The year to rank (default: the last with data) n How many units to show at each end title The figure and table title Returns ------- ExtremesResult A result with ``.df`` (the ``n`` highest and the ``n`` lowest units: ``geo_id``, ``name``, the group columns of the table, the variable, ``rank`` with 1 for the highest, and ``end``: ``"highest"`` or ``"lowest"``), ``.fig`` (horizontal bars, the two ends in different colours), ``.gt``, ``.highest``, ``.lowest``, ``.median``, ``.n_units`` (units with data), ``.year``, ``.notes`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo mpi = geo.load("sdg01", variables="sdg1_2_mpi", groups=["department"]) res = geo.extremes(mpi, "sdg1_2_mpi", year=2024, n=5) print(res.interpret()) res.fig ``` ## geo.distribution(data, variable: 'str', year=None, *, by: 'str | None' = None, kind: 'str' = 'histogram', bins: 'int' = 30, title: 'str | None' = None, ref: 'str | None' = None) -> 'DistributionResult' The distribution of a variable across units, per year or per group. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` variable The numeric column to describe year A year, a list of years or ``"all"``. By default (``None``) every year is shown next to the others, with one exception: a histogram overlays the years only when the panel has up to four of them, and otherwise shows the last year with data. With ``by``, one year is used: the last with data by default, or every year pooled with ``"all"`` by Compare groups instead of years: a column of the table, or a coarser scale of its units (e.g. the first-level administrative scale), whose names are added kind ``"histogram"`` (overlaid, on common bins), ``"box"`` or ``"violin"`` (one per year or group) or ``"ridgeline"`` (one density per year or group, stacked) bins Number of bins of the histogram title The figure title Returns ------- DistributionResult A result with ``.df`` (one row per year, or per group of ``by``: ``n``, ``mean``, ``sd``, ``min``, ``p25``, ``median``, ``p75``, ``max`` and ``skewness``, the sample-adjusted coefficient that is zero for a symmetric distribution), ``.fig``, ``.kind``, ``.year`` (when a single year is shown), ``.notes`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.distribution("sdg01", "sdg1_2_mpi", by="department", kind="box") print(res.interpret()) res.fig ``` ## geo.scatter(data, x: 'str', y: 'str', year=None, *, color: 'str | None' = None, size: 'str | None' = None, fit: 'str | None' = 'ols', log_x: 'bool' = False, log_y: 'bool' = False, title: 'str | None' = None, ref: 'str | None' = None) -> 'ScatterResult' Two variables against each other in one year, one point per unit. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` x, y The numeric columns on the horizontal and the vertical axis year The year to plot (default: the last with data for ``x``), or ``"all"`` to pool every year (one point per unit and year) color Colour the points by a column (categories, or a numeric variable on a continuous scale) or by a coarser scale of the units, whose names are added size A non-negative numeric column that sets the area of the points fit The line drawn: ``"ols"`` (least squares, with its slope, R² and the correlation written on the figure), ``"lowess"`` (a local smooth) or ``None`` log_x, log_y Logarithmic axes; the line and the numbers then use natural logarithms, and units with a value of zero or less are left out title The figure title Returns ------- ScatterResult A result with ``.df`` (the units plotted: ``geo_id``, ``name`` and the columns used), ``.fig`` (the unit names show on hover), ``.slope``, ``.intercept``, ``.r2`` and ``.p_value`` (of the least-squares line of ``y`` on ``x``, whichever ``fit`` is drawn), ``.corr`` (Pearson), ``.n`` (units), ``.year``, ``.notes`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.scatter("sdg", "urbanization", "index_sdg", color="department", size="population") print(res.interpret()) res.fig ``` ## geo.trend(data, variable: 'str', *, by: 'str | None' = None, units: 'bool' = True, stat: 'str' = 'mean', max_units: 'int' = 400, title: 'str | None' = None, ref: 'str | None' = None) -> 'TrendResult' A variable over time: its mean (or median) across units, with a band, and each unit's path. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable The numeric column to follow by One line per group: a column of the table, or a coarser scale of its units (e.g. the first-level administrative scale), whose names are added units Draw each unit's own trajectory as a thin grey line behind the summary stat ``"mean"`` (band: one standard error on each side) or ``"median"`` (band: the interquartile range) max_units Draw at most this many unit trajectories (a reproducible random sample when there are more); the statistics always use every unit title The figure title Notes ----- The panel needs at least two years with data. Returns ------- TrendResult A result with ``.df`` (one row per year and group: the statistic in a column named ``mean`` or ``median``, the band in ``lower`` and ``upper``, and ``n`` units), ``.fig``, ``.first`` and ``.last`` (years), ``.value_first``, ``.value_last`` and ``.change`` (the statistic across all units), ``.share_rising`` (share of units with data in both years that ended higher), ``.n_units``, ``.notes`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.trend("ntl", "ln_NTLpc") print(res.interpret()) res.fig ``` ## geo.map(data, variable: 'str', year=None, scale: 'str | None' = None, title: 'str | None' = None, simplify: 'float | str | None' = 'auto', ref: 'str | None' = None, *, scheme: 'str | None' = None, k: 'int' = 5, bins=None, change: 'bool' = False, **kwargs) An interactive choropleth map (Plotly) of one variable: in one year, in several years side by side, or its change between two years; on a continuous colour scale or in classes. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (with or without geometry), e.g. from ``geo.load``, ``geo.change`` or ``geo.aggregate`` variable The column to colour by year The year to map (default: the last year with data), or a list of years (``[2012, 2024]``): one map per year, side by side, in the same classes or on the same colour range, computed over all the years shown, so the maps can be compared. Ignored for tables without a ``year`` column scale The units' scale (default: inferred from ``geo_id``) title The figure title (default: the variable's label and the year) simplify How much to simplify the drawn boundaries, which keeps the figure light. ``"auto"`` (the default) removes detail smaller than about 1/600 of the map's extent, invisible at the scale of the whole map; a number is a tolerance in degrees (0.005 is about 500 m), for maps you will zoom into; ``None`` draws the boundaries in full detail. Neighbouring units keep fitting each other scheme Draw classes instead of a continuous scale, one colour per class with its range in the legend: ``"fisher_jenks"`` (natural breaks: similar values together), ``"quantiles"`` (the same number of units in each class) or ``"equal_interval"`` (classes of the same width) k The number of classes of a ``scheme`` bins Explicit class limits in increasing order (``[20, 40, 60]``), instead of a ``scheme``; values below the first or above the last limit get a class of their own change Map the change of the variable (last minus first year, as ``geo.change`` computes it) between the first and last year with data, or between the two years given in ``year``, on a diverging scale centred at zero. With a ``scheme``, the classes split at zero: they are computed within the falls and within the rises **kwargs For the continuous map of one year, passed to ``plotly.express.choropleth_map`` (e.g. ``color_continuous_scale="Viridis"``, ``range_color=(0, 100)``). With classes, several years or ``change``, the options are ``color_continuous_scale``, ``range_color``, ``map_style``, ``opacity``, ``center``, ``zoom``, ``height`` and ``width`` Returns ------- plotly.graph_objects.Figure A ``plotly.graph_objects.Figure``; it displays in a notebook, and ``.show()`` displays it from a script or a cell that does more afterwards. A map in classes keeps their limits in ``fig.layout.meta["bins"]``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo # Multidimensional poverty in both periods, in common natural-break classes geo.map("sdg01", "sdg1_2_mpi", year=[2012, 2024], scheme="fisher_jenks") ``` ## geo.weights(data, method: 'str' = 'queen', *, k: 'int' = 6, threshold: 'float | None' = None, power: 'float' = 1.0, row_standardize: 'bool' = True, attach_islands: 'bool' = True, gdf=None, ref: 'str | None' = None) Spatial weights for the units of a scale, a dataset or a panel. Parameters ---------- data A scale id (every unit of that scale), a dataset id or a panel (the units with data), or a GeoDataFrame with ``geo_id`` method ``"queen"`` (units sharing a border or a corner), ``"rook"`` (a border), ``"knn"`` (the ``k`` nearest), ``"distance_band"`` (within ``threshold`` metres) or ``"inverse_distance"`` (within the same band, weighted by distance ** -``power``) k Neighbours for ``"knn"`` threshold Distance in metres for ``"distance_band"`` (default: the smallest distance at which every unit has a neighbour) power Distance decay for ``"inverse_distance"`` row_standardize Make each unit's weights sum to one attach_islands Link a unit without contiguous neighbours to its nearest unit gdf Boundaries to use instead of the project's Returns ------- libpysal.weights.W A ``libpysal.weights.W`` keyed by ``geo_id``, to pass as ``w=`` to the spatial functions. They build queen weights themselves when ``w`` is not given. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo w = geo.weights("municipality") # queen contiguity, 343 units w6 = geo.weights("municipality", method="knn", k=6) # the six nearest print(w.n, round(w.mean_neighbors, 2), w6.n, w6.mean_neighbors) geo.moran("sdg", "index_sdg", w=w6).fig ``` ## geo.connectivity(data, w=None, *, method: 'str' = 'queen', tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None) A map of the neighbour links of a set of weights, with the distribution of neighbours per unit. Parameters ---------- data A scale id, a dataset id or a panel, or a GeoDataFrame with ``geo_id`` w The weights to draw (default: ``method`` weights built from the boundaries) method The weights to build when ``w`` is not given (see ``weights``) tiles The basemap style (``None``: no basemap) title The figure title gdf Boundaries to use instead of the project's Returns ------- result object A result with ``.fig`` (the links over the map), ``.fig_hist`` (neighbours per unit), ``.df`` (one row per unit with its number of neighbours), the weights ``.w`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.connectivity("municipality") print(res.interpret()) res.fig ``` ## geo.moran(data, variable: 'str', year=None, *, w=None, permutations: 'int' = 999, seed: 'int | None' = 12345, title: 'str | None' = None, gdf=None, ref: 'str | None' = None) Global Moran's I of a variable: is it clustered in space? Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` variable The numeric column to test year The year to test (default: the last with data), or ``"all"`` for Moran's I in every year, on the units that have data in all of them w Spatial weights from ``weights`` (default: queen contiguity among the units with data) permutations Random permutations behind the pseudo p-value seed Makes the permutations reproducible (``None``: leave the random state alone) title The figure title gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``) Returns ------- result object For one year, a result with ``.moran_i``, ``.p_sim``, ``.z_sim``, ``.fig`` (the Moran scatterplot: each unit's standardized value against its neighbours' mean), ``.df`` (one row per unit with its quadrant) and ``.interpret()``. For ``year="all"``, a result with ``.df`` (one row per year: ``moran_i``, ``p_sim``, ``z_sim``), ``.fig`` (Moran's I over time) and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.moran("sdg", "index_sdg", year=2024) print(res.interpret()) print(geo.moran("sdg", "index_sdg", year="all").df) # Moran's I in each year res.fig ``` ## geo.lisa(data, variable: 'str', year=None, *, w=None, permutations: 'int' = 999, seed: 'int | None' = 12345, alpha: 'float' = 0.05, panels: 'str' = 'both', tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None) Local Moran (LISA) clusters of a variable: where are the hot spots, cold spots and outliers? The figure shows the Moran scatterplot (each unit's standardized value against its neighbours' mean, coloured by cluster, grey where not significant) next to the map of the clusters. Both are interactive, and a click on a cluster type in the legend shows or hides it in both. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` variable The numeric column to analyse year The year to analyse (default: the last with data) w Spatial weights from ``weights`` (default: queen contiguity among the units with data) permutations Random permutations behind each unit's pseudo p-value seed Makes the permutations reproducible alpha Significance level below which a unit is assigned to a cluster panels What ``.fig`` shows: ``"both"`` (the scatterplot and the map side by side), ``"map"`` (the cluster map) or ``"scatter"`` (the Moran scatterplot) tiles The basemap style (``None``: no basemap) title The figure title gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``) Returns ------- result object A result with ``.fig`` (by default the Moran scatterplot and the cluster map side by side; see ``panels``), ``.fig_map`` (the cluster map alone), ``.fig_scatter`` (the Moran scatterplot alone), ``.df`` (one row per unit: local statistic, p-value and ``cluster``: High-High, Low-Low, High-Low, Low-High or not significant), the number of units of each type (``.n_hh``, ``.n_ll``, ``.n_hl``, ``.n_lh``, ``.n_ns``), the global ``.moran_i`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.lisa("sdg", "index_sdg", year=2024) print(res.interpret()) res.fig ``` ## geo.beta_convergence(data, variable: 'str', controls=None, *, first=None, last=None, model: 'str' = 'ols', w=None, fixed_effects=None, vcov: 'str' = 'hetero', draws: 'int' = 10000, seed: 'int' = 12345, min_obs: 'int' = 10, title: 'str | None' = None, gdf=None, ref: 'str | None' = None) Beta-convergence: did the units that started lower grow faster between two years? Regresses each unit's average yearly log growth between ``first`` and ``last`` on the logarithm of its level in ``first``. A negative slope is convergence. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable A numeric, strictly positive column in levels (its logarithm is taken here, so do not pass a variable that is already a logarithm) controls Numeric column(s) that enter at their value in ``first`` (conditional convergence) first, last The two years (default: the first and the last year with data) model ``"ols"``, or a spatial model estimated on the boundaries' weights: ``"sar"`` (spatial lag of growth), ``"sem"`` (spatial error), ``"slx"`` (spatial lags of the regressors) or ``"sdm"`` (spatial Durbin: both lags) w Spatial weights from ``weights`` for the spatial models (default: queen contiguity among the units with data in both years) fixed_effects Column(s), or coarser scale(s) the units lie in (e.g. the scale above them), that enter as dummy variables vcov Standard errors of the OLS estimate: ``"hetero"`` (robust, HC1) or ``"iid"`` draws Simulation draws behind the standard errors of the spatial models' impacts seed Makes the draws reproducible min_obs The fewest units the regression accepts title The figure title gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``). Boundaries are loaded only for the spatial models; with ``"ols"``, passing ``gdf`` adds the growth map Returns ------- result object A result with ``.beta_total`` (the slope; for the spatial models the total impact, next to ``.beta_direct`` and ``.beta_indirect``, the spillover), their standard errors ``.se_total``, ``.se_direct`` and ``.se_indirect``, ``.speed`` (of convergence, per year) and ``.half_life`` (years to close half of a gap), ``.rho`` and ``.lam`` (the spatial parameters), ``.r2``, ``.aic``, ``.n_obs``, ``.horizon`` (years between ``first`` and ``last``), ``.df`` (one row per unit: ``initial``, ``final``, ``growth`` and the controls), ``.fig`` (growth against the initial log level, with the fitted line), ``.fig_conditional`` (the same with controls and fixed effects partialled out; ``None`` without them), ``.fig_map`` (the map of growth; ``None`` without boundaries), ``.gt`` and ``.summary`` (the estimates; the OLS column is always there), ``.impacts`` (every regressor's impacts, spatial models), ``.w_spec``, ``.notes`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.beta_convergence("sdg", "index_sdg") # ~2012 to ~2024 print(res.interpret()) res.fig ``` ## geo.sigma_convergence(data, variable: 'str', *, first=None, last=None, vcov: 'str' = 'hetero', title: 'str | None' = None, ref: 'str | None' = None) Sigma-convergence: does the dispersion of a variable across units fall over time? Measures the dispersion of the logarithm of ``variable`` in every year, on the units observed in all of them, and estimates its trend. A negative trend is convergence. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable A numeric, strictly positive column in levels (its logarithm is taken here) first, last The first and last year to use (default: every year with data). The trend needs at least three years vcov Standard errors of the trend: ``"hetero"`` (robust, HC1) or ``"iid"`` title The figure title Returns ------- result object A result with ``.std_slope`` (the trend of the log standard deviation per year: negative when dispersion falls), ``.std_se``, ``.std_pvalue`` and ``.std_r2``, the same for the Gini index (``.gini_slope``...) and the coefficient of variation (``.cv_slope``...), ``.n_units``, ``.n_periods``, ``.df`` (one row per year: ``n_units``, ``mean``, ``std``, ``gini`` and ``cv`` of the logarithm), ``.fig`` (dispersion over time with the fitted trends), ``.gt`` and ``.summary`` (one row per measure), ``.notes`` (among them the units left out for lack of data) and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.sigma_convergence("pop", "pop") # 20 years, 333 units print(res.interpret()) res.fig ``` ## geo.convergence_clubs(data, variable: 'str', *, first=None, last=None, log: 'bool' = False, hp_filter: 'bool' = True, hp_lambda: 'float' = 400.0, trim: 'float' = 0.3, tcrit: 'float' = -1.65, cr: 'float' = 0.0, increment: 'float' = 0.05, max_cr: 'float' = 3.0, fraction: 'float' = 0.0, adjust: 'bool' = False, merge: 'str' = 'ps', map: 'bool' = True, tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None) Convergence clubs (Phillips and Sul): do all units converge, or do groups of them? Tests whether the units' paths converge (the log t test) and, when they do not, sorts the units into clubs that converge among themselves, on the units observed in every year. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable The numeric column to analyse, used as given: pass a logarithm, or set ``log`` first, last The first and last year to use (default: every year with data). The years in between must all be there: at least 6 of them at the default ``trim`` log Take the logarithm of ``variable`` first (it must then be strictly positive) hp_filter Analyse each unit's Hodrick-Prescott trend instead of its raw path hp_lambda The filter's smoothing (400 for yearly data) trim The share of the first years that the log t regression leaves out tcrit The t-statistic above which convergence is not rejected (-1.65: the 5% level) cr, increment, max_cr The threshold for adding a unit to a club's core group, the step by which it is raised while the club fails the test, and its ceiling fraction Sort the units by their mean over the last ``1 - fraction`` of the years instead of the last year (0) adjust Add the units to a club one at a time (Schnurbus et al.) instead of raising ``cr`` merge Join neighbouring clubs that converge together: ``"ps"`` (until none do), ``"single"`` (one pass) or ``"none"`` map Draw the map of the clubs (``False``: no boundaries are loaded) tiles The basemap style (``None``: no basemap) title The figure title gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``) Returns ------- result object A result with ``.n_clubs``, ``.n_divergent`` (units in no club), ``.converged`` (all units form one club), ``.global_tstat`` and ``.global_beta`` (the log t test on all units), ``.n_units``, ``.n_periods``, ``.membership`` (one row per unit with its ``club``; 0 = in no club), ``.summary`` and ``.gt`` (one row per club with its test), ``.df`` (one row per unit and year: ``value``, the ``relative`` path and ``club``), ``.fig`` (the clubs' average paths), ``.fig_paths`` (every unit's path), ``.fig_clubs`` (one panel per club), ``.fig_map`` (the map of the clubs; ``None`` without boundaries), ``.notes`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.convergence_clubs("ntl", "ln_NTLpc") # 2012-2020, 316 units print(res.interpret()) res.fig_map ``` ## geo.transitions(data, variable: 'str', *, k: 'int' = 5, scheme: 'str' = 'quantiles', bins=None, per_year: 'bool' = True, relative: 'bool' = False, title: 'str | None' = None, ref: 'str | None' = None) Markov transitions between classes of a variable's distribution: how mobile are the units? Every unit is assigned to one of ``k`` classes in each year, and all the moves from one year to the next are counted in a transition matrix: the probability of being in each class next year, given this year's class. Needs the ``[dynamics]`` extra. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable The numeric column to follow, observed in at least two years; the units without a value in every year are left out k Number of classes scheme How classes are formed: ``"quantiles"`` (equal numbers of units), ``"equal_interval"`` or ``"fisher_jenks"`` bins Upper bounds of fixed classes, the same in every year (replaces ``k``, ``scheme`` and ``per_year``) per_year Classify each year on its own, so a class is a position within that year's distribution (default); ``False`` classifies all years together, so classes are fixed levels relative Divide the values by each year's mean first title The figure title Returns ------- result object A result with ``.p`` (the transition probabilities, rows = this year's class) and ``.counts``, ``.fig`` (the matrix as a heatmap), ``.gt`` (mobility indices, long-run shares and expected stays), ``.df`` (one row per unit and year with its ``state``), ``.states``, ``.steady_state`` (the long-run distribution), ``.sojourn`` (expected years in each class), the mobility indices ``.shorrocks``, ``.prais`` and ``.bartholomew``, ``.n_transitions`` and ``.interpret()``. Example: ```python !pip install -q "geo4bolivia[dynamics]" import geo4bolivia as geo res = geo.transitions("ntl", "ln_NTLpc", k=5) print(res.interpret()) res.fig ``` ## geo.spatial_transitions(data, variable: 'str', *, w=None, k: 'int' = 5, m: 'int | None' = None, per_year: 'bool' = False, relative: 'bool' = True, title: 'str | None' = None, gdf=None, ref: 'str | None' = None) Spatial Markov transitions: does a unit's mobility depend on where its neighbours stand? Rey's spatial Markov chain splits the transition matrix of ``transitions`` by the class of each unit's neighbours (their weighted mean), and tests whether the matrices differ. Needs the ``[dynamics]`` extra. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable The numeric column to follow, observed in at least two years; the units without a value in every year are left out w Spatial weights from ``weights`` (default: queen contiguity among the units with data) k Number of quantile classes of the variable m Number of quantile classes of the neighbours' mean (default: ``k``) per_year Classify each year on its own; by default all years are classified together, so classes are fixed levels relative Divide the values by each year's mean first (default), so that classes are levels relative to the average unit; needs positive means title The figure title gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``) Returns ------- result object A result with ``.p_global`` (the transition probabilities of all units), ``.p_conditional`` (one matrix per class of the neighbours, from the lowest to the highest), ``.steady_states`` (the long-run distribution in each neighbourhood), ``.fig`` (the conditional matrices side by side), ``.gt`` (the tests of equal dynamics across neighbourhoods), ``.lr_stat``, ``.lr_p``, ``.q_stat``, ``.q_p``, ``.dof``, ``.df`` (one row per unit and year with its ``state`` and ``neighbor_state``) and ``.interpret()``. Example: ```python !pip install -q "geo4bolivia[dynamics]" import geo4bolivia as geo res = geo.spatial_transitions("ntl", "ln_NTLpc", k=5) print(res.interpret()) res.fig ``` ## geo.lisa_transitions(data, variable: 'str', first=None, last=None, *, w=None, permutations: 'int' = 999, seed: 'int | None' = 12345, alpha: 'float' = 0.05, tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None) Transitions between local Moran (LISA) types: did the hot spots and cold spots persist? Runs ``lisa`` in two years on the same units and the same weights, and counts how the units moved between the types High-High, Low-High, Low-Low, High-Low and Not significant. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable The numeric column to analyse first, last The two years to compare (default: the first and last with data); only the units with a value in both are used w Spatial weights from ``weights`` (default: queen contiguity among the units with data) permutations Random permutations behind each unit's pseudo p-value seed Makes the permutations reproducible alpha Significance level below which a unit is assigned to a cluster tiles The basemap style of the map (``None``: no basemap) title The title of the matrix figure gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``) Returns ------- result object A result with ``.df`` (one row per unit: ``cluster_``, ``cluster_``, ``moved`` and the ``transition`` in words), ``.matrix`` (units by type in the first year, rows, and in the last year, columns; always the five types, in the same order), ``.shares`` (each row as shares of its total; empty rows are missing), ``.fig`` (the matrix as a heatmap: counts in the cells, coloured by the share of the row), ``.fig_map`` (the types in the last year, with the units that changed type outlined), ``.n_units``, ``.n_stayed``, ``.n_moved``, ``.share_stayed``, the global Moran's I of each year (``.moran_i_first``, ``.moran_i_last``, with ``.p_sim_first`` and ``.p_sim_last``) and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.lisa_transitions("sdg", "index_sdg") # ~2012 to ~2024 print(res.interpret()) res.fig ``` ## geo.rank_mobility(data, variable: 'str', first=None, last=None, *, k: 'int' = 5, top: 'int' = 10, ascending: 'bool' = False, title: 'str | None' = None, ref: 'str | None' = None) Rank mobility between two years: how much did the units change position? Ranks the units in each year, compares the two rankings (rank correlations, the change of position of every unit) and follows the units between the quantile groups of the two years. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable The numeric column to rank first, last The two years to compare (default: the first and last with data); only the units with a value in both are ranked k Number of quantile groups of the transition matrix (``Q1``: the lowest values) top Number of units in ``.climbers`` and ``.fallers`` ascending ``False`` (default): rank 1 is the highest value; ``True``: the lowest, for variables where less is better title The title of the rank figure Returns ------- result object A result with ``.df`` (one row per unit, ordered by the last rank: ``_``, ``_``, ``rank_``, ``rank_``, ``rank_change``, positive when the unit climbed, and its quantile ``group_`` in each year), ``.fig`` (the rank in the first year against the rank in the last, with the line of no change), ``.matrix`` (the shares of each group of the first year, rows, found in each group of the last year, columns; rows sum to one), ``.counts`` and ``.fig_matrix`` (its heatmap), ``.climbers`` and ``.fallers`` (the ``top`` units by rank change), ``.gt`` (both as one table), ``.spearman`` and ``.kendall`` (rank correlations of the two years), ``.mean_abs_change`` and ``.median_abs_change`` (positions moved by a unit), ``.share_same_group``, ``.n_units``, ``.n_climbed``, ``.n_fell`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.rank_mobility("sdg", "index_sdg") print(res.interpret()) res.fig ``` ## geo.inequality(data, variable: 'str', *, measures=('gini', 'theil'), first=None, last=None, spatial: 'bool' = False, w=None, permutations: 'int' = 99, lorenz: 'bool' = True, title: 'str | None' = None, gdf=None, ref: 'str | None' = None) -> 'InequalityResult' Inequality between units in each year: is it narrowing or widening? Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable The numeric column to measure, in levels (not logs) measures ``"gini"``, ``"theil"`` (needs strictly positive values) and/or ``"cv"`` (the coefficient of variation: standard deviation over the mean) first, last The first and the last year to use (default: every year with data); the years between them are used too spatial Add the spatial Gini: the part of the Gini owed to pairs of neighbouring units, with a permutation test. It needs boundaries and weights, and uses the units with data in every year. Giving ``w`` implies it w Spatial weights from ``weights`` (default: queen contiguity among the units with data) permutations Random permutations behind the spatial Gini's pseudo p-value (0: no test) lorenz Also draw the Lorenz curve title The title of the main figure gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``) Returns ------- InequalityResult A result with ``.df`` (one row per year: ``n_units`` and one column per measure, plus ``gini_spatial`` and ``gini_spatial_p`` when spatial), ``.fig`` (the measures over time), ``.summary`` (one row per measure: its value in the ``first`` and ``last`` year, their ``change``, and the ``slope``, ``se`` and ``pvalue`` of its log-trend, which needs three or more years), ``.gt`` (the trends; with two years, the measures in both years), ``.lorenz`` (one row per unit and year, from the lowest value to the highest, with the cumulative share of units ``cum_units`` and of the variable ``cum_share``), ``.fig_lorenz`` (one curve per year, the first and the last when there are more than four, with the line of equality), ``.lorenz_share(year, share)``, ``.n_years``, ``.n_units``, ``.w_spec``, ``.notes`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.inequality("sdg", "index_sdg") print(res.interpret()) res.fig_lorenz ``` ## geo.theil_decomposition(data, variable: 'str', by: 'str | None' = None, *, permutations: 'int' = 0, seed: 'int' = 12345, title: 'str | None' = None, ref: 'str | None' = None) -> 'TheilResult' Theil's index split in two: inequality between groups of units and inequality within them. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` variable The numeric column to measure, in levels and strictly positive by The groups: a column of the table, or a coarser scale the units nest in (see ``ancestors``). Default: the coarsest such scale with two or more units. A unit must stay in the same group in every year permutations Random reassignments of units to groups behind the pseudo p-value of the between component (0: no test) seed Makes the permutations reproducible title The figure title Returns ------- TheilResult A result with ``.df`` (one row per year: ``theil``, ``between``, ``within``, which add up to it, ``between_share``, and ``p_between`` with permutations), ``.fig`` (the two components stacked, with the between share), ``.gt``, ``.by``, ``.n_groups``, ``.notes`` and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.theil_decomposition("sdg", "index_sdg", by="department") print(res.interpret()) res.fig ``` ## geo.regression(data, y: 'str', x, year=None, *, fixed_effects=None, cluster: 'str | None' = None, robust: 'bool' = True, title: 'str | None' = None, ref: 'str | None' = None) -> 'RegressionResult' Ordinary least squares, with several specifications side by side. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` or ``combine`` y The outcome column x The regressors: a column or a list of columns (one model), or a list of lists (one model per list, e.g. adding controls step by step) year The year to use (default: the last with data for ``y`` and every regressor), a list of years, or ``"all"`` to pool every year fixed_effects A column, a coarser scale (e.g. the scale above the units'), ``"year"``, or a list of them: absorbed as dummies and reported as Yes/No rows instead of coefficients. A list with one entry per model, where an entry is ``None`` or a list, sets them model by model cluster A column or a coarser scale: standard errors clustered by it (p-values and intervals then use a t distribution with one degree of freedom less than clusters) robust Heteroskedasticity-robust standard errors (HC1) when there is no ``cluster``; ``False``: classical standard errors title The title of the table and of the figure Returns ------- RegressionResult A result with ``.gt`` (the regression table: coefficients with stars, standard errors in parentheses, the fixed effects, N and R²), ``.fig`` (the coefficients with 95% confidence intervals across models), ``.df`` (tidy: ``model``, ``term``, ``estimate``, ``std_error``, ``statistic``, ``p_value``, ``conf_low``, ``conf_high``), ``.summary`` (one row per model: ``n``, ``r2``, ``r2_adj``, ``fixed_effects``, ``std_errors``), ``.models`` (the fitted statsmodels results) and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.regression("sdg", "index_sdg", [["urbanization"], ["urbanization", "population"]], fixed_effects=[None, "department"]) print(res.interpret()) res.gt ``` ## geo.panel_model(data, y: 'str', x, *, models=('pooled', 'between', 'fe', 're'), time_effects: 'bool' = False, cluster: 'bool' = True, title: 'str | None' = None, ref: 'str | None' = None) -> 'PanelModelResult' The standard panel estimators side by side, and the Hausman test of fixed against random effects. The units are ``geo_id`` and the periods ``year``; every year with data is used. Parameters ---------- data A dataset id, or a panel with ``geo_id`` and ``year``, e.g. from ``load`` or ``combine`` y The outcome column x The regressors: a column or a list of columns models The estimators to compare, in order: ``"pooled"`` (least squares on all rows), ``"between"`` (on the units' averages over the years), ``"fe"`` (fixed effects: each unit's deviations from its own average) and ``"re"`` (random effects) time_effects Add year effects (to every estimator but ``"between"``) cluster Standard errors clustered by unit; ``False``: heteroskedasticity-robust. The between estimator, with one row per unit, is always heteroskedasticity-robust title The title of the table and of the figure Returns ------- PanelModelResult A result with ``.gt`` (the comparison table), ``.fig`` (the coefficients with 95% confidence intervals across estimators), ``.df`` (tidy: ``model``, ``term``, ``estimate``, ``std_error``, ``statistic``, ``p_value``, ``conf_low``, ``conf_high``), ``.summary`` (one row per model: ``n``, ``units``, ``years``, ``r2``, ``r2_within``, ``r2_between``, ``r2_overall``), ``.models`` (the fitted linearmodels results, keyed by estimator), ``.hausman`` (``statistic``, ``df``, ``p_value``; ``None`` unless both ``"fe"`` and ``"re"`` are estimated and the test can be computed) and ``.interpret()``. The Hausman test compares the fixed- and random-effects coefficients of the regressors with classical covariance matrices that share the residual variance of the fixed-effects model, so that their difference is positive semi-definite. Needs the ``linearmodels`` library (the ``[panel]`` extra). Example: ```python !pip install -q "geo4bolivia[panel]" import geo4bolivia as geo panel = geo.combine(["ntl", "pop"], scale="municipality") res = geo.panel_model(panel, "ln_NTLpc", "pop_density") print(res.interpret()) res.gt ``` ## geo.spatial_model(data, y: 'str', x, year=None, *, model: 'str' = 'durbin', method: 'str' = 'ml', w=None, fixed_effects: 'str | None' = None, impacts: 'bool' = True, n_draws: 'int' = 10000, seed: 'int | None' = 12345, title: 'str | None' = None, gdf=None, ref: 'str | None' = None) A spatial regression for one year, with direct, indirect (spillover) and total impacts. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` or ``combine`` y The outcome column x The regressors: a column or a list of columns year The year to model (default: the last with data for ``y`` and every regressor) model ``"ols"`` (no spatial term), ``"lag"`` (neighbours' outcome), ``"error"`` (spatially correlated errors), ``"slx"`` (neighbours' regressors), ``"durbin"`` (neighbours' outcome and regressors) or ``"durbin_error"`` (neighbours' regressors and spatially correlated errors) method ``"ml"`` (maximum likelihood) or ``"gm"`` (generalized method of moments; not for ``"durbin_error"``) w Spatial weights from ``weights`` (default: queen contiguity among the units with data) fixed_effects A column or a coarser scale, added as dummies impacts Compute the direct, indirect and total impacts where the model has them n_draws Simulation draws behind the standard errors of the impacts seed Makes the draws reproducible title The title of the table gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``) Returns ------- result object A result with ``.df`` (tidy coefficients: ``variable``, the column name, with ``const`` for the intercept and ``W_`` for a spatial lag; ``term``, its label; ``estimate``, ``se``, ``z``, ``p``), ``.gt`` (the coefficient table), ``.impacts`` (one row per regressor: ``variable``, ``term``, ``direct``, ``indirect``, ``total`` and their standard errors; ``None`` for ``"ols"`` and ``"error"``), the spatial parameters ``.rho`` (lag) and ``.lam`` (error), ``.r2``, ``.aic``, ``.log_likelihood``, ``.n_obs``, ``.model_obj`` (the fitted spreg model) and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.spatial_model("sdg", "index_sdg", "urbanization", model="durbin") print(res.interpret()) res.gt ``` ## geo.spatial_diagnostics(data, y: 'str', x, year=None, *, w=None, fixed_effects: 'str | None' = None, alpha: 'float' = 0.05, gdf=None, ref: 'str | None' = None) Tests for spatial dependence left in a least-squares regression, and the spatial model they point to. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` or ``combine`` y The outcome column x The regressors: a column or a list of columns year The year to test (default: the last with data for ``y`` and every regressor) w Spatial weights from ``weights`` (default: queen contiguity among the units with data) fixed_effects A column or a coarser scale, added as dummies alpha Significance level of the decision rule gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``) Returns ------- result object A result with ``.df`` (one row per test: Moran's I of the residuals, the Lagrange multiplier tests for a lag and for an error model, their robust forms and the joint test, with ``statistic``, ``df`` and ``p``), ``.gt`` (the same as a table), ``.recommendation`` (``"ols"``, ``"lag"`` or ``"error"``), ``.reasoning``, ``.moran_i_resid``, ``.ols_model`` (the fitted spreg model) and ``.interpret()``. Example: ```python !pip install -q geo4bolivia import geo4bolivia as geo res = geo.spatial_diagnostics("sdg", "index_sdg", "urbanization") print(res.interpret()) res.gt ``` ## geo.mgwr(data, y: 'str', x, year=None, *, kernel: 'str' = 'bisquare', criterion: 'str' = 'AICc', max_iter: 'int' = 200, tiles: 'str | None' = 'carto-positron', title: 'str | None' = None, gdf=None, ref: 'str | None' = None) Multiscale geographically weighted regression: where, and at which scale, does ``x`` relate to ``y``? A regression of ``y`` on ``x`` is fitted at every unit, weighting the other units by the distance between centroids, so each coefficient becomes a surface that can be mapped. MGWR finds one bandwidth per term: a bandwidth close to the number of units means an association that is about the same everywhere (a global process), a small one an association that changes from place to place (a local process). It follows the workflow of the mgwr library (Oshan et al. 2019): the variables are z-standardized, so the coefficients are in standard deviations and comparable; the bandwidths are the numbers of nearest units, found by back-fitting; and each term's local t-tests are corrected for multiple testing at the 5% level. The search has a floor: the mgwr library does not try bandwidths below about 40 nearest units plus two per term, so a bandwidth at that floor means "as local as the search goes", not exactly that many neighbours. It is slow: the search fits the model again and again, which takes from half a minute to a minute and a half with several hundred units and one or two variables, and longer with more of either. Parameters ---------- data A dataset id, or a panel with ``geo_id`` (and ``year``), e.g. from ``load`` or ``combine`` y The numeric column to explain x The explanatory column, or a list of them year The year to analyse (default: the last with data for ``y`` and ``x``) kernel How weights fall with distance: ``"bisquare"``, ``"gaussian"`` or ``"exponential"`` criterion What the bandwidth search minimises: ``"AICc"``, ``"AIC"``, ``"BIC"`` or ``"CV"`` max_iter The most rounds of the bandwidth search (fewer is faster and rougher) tiles The basemap style (``None``: no basemap) title The title of the residual map; each coefficient map adds its variable gdf Boundaries to use instead of the project's (a GeoDataFrame with ``geo_id``) Returns ------- result object A result with ``.bw`` (the bandwidth of each term, in nearest units) and ``.bw_interval`` (its 95% interval); ``.df`` (one row per unit: ``_coef``, ``_se``, ``_t`` and ``_significant`` for the intercept ``const`` and each column of ``x``, the ``residual`` and ``local_cn``, the local condition number, which signals local collinearity above about 30); ``.figs`` (a map of the local coefficients per term, by column name, with the units that are not significant in grey); ``.fig`` (the residual map); ``.gt`` (the summary table) and ``.gt_bw`` (the bandwidth table); ``.r2``, ``.aicc``, ``.adj_alpha`` and ``.critical_t`` (per term), ``.notes`` and ``.interpret()``. Example: ```python !pip install -q "geo4bolivia[mgwr]" import geo4bolivia as geo res = geo.mgwr("sdg", "index_sdg", "urbanization") # about half a minute print(res.interpret()) res.figs["urbanization"] ``` ## geo.for_expdpy(data, scale: 'str | None' = None, years=None, variables=None, labels: 'dict | None' = None, groups=None, ref: 'str | None' = None) ``(df, df_def)`` in the format expdpy expects. Parameters ---------- data A dataset id or your own panel with ``geo_id`` and ``year`` scale, years, variables As in ``geo.load`` (for a dataset id) labels ``{variable: description}`` for your own columns groups Coarser scales whose unit names to add as columns Notes ----- These are the parameters of ``for_geometrics``. ``groups`` is handy for expdpy's by-group views, e.g. ``groups=["department"]`` adds a ``department`` column (the department's name) for ``ex.explore_trend_plot_by_group``. Returns ------- dict df : the panel with ``geo_id``, ``name``, ``year``, the group names and the variables df_def : the data dictionary (``var_name``, ``var_def``, ``label``, ``type``, ``role``, ``can_be_na``); pass both to ``ex.set_labels(df, df_def, set_panel=True)`` Example: ```python !pip install -q geo4bolivia expdpy import geo4bolivia as geo import expdpy as ex df, df_def = geo.for_expdpy("sdg", variables="index_sdg", groups=["department"]) df = ex.set_labels(df, df_def, set_panel=True) ex.explore_trend_plot_by_group(df, group_var="department", var="index_sdg").fig ``` ## geo.for_geometrics(data, scale: 'str | None' = None, years=None, variables=None, labels: 'dict | None' = None, groups=None, ref: 'str | None' = None) ``(gdf, df, df_dict)`` in the format geometrics expects. Parameters ---------- data A dataset id (see ``geo.datasets()``) or your own panel with ``geo_id`` and ``year`` (from ``geo.link``, ``geo.combine``, ``geo.zonal``...) scale, years, variables As in ``geo.load`` (for a dataset id) labels ``{variable: description}`` for your own columns (dataset variables are described from the metadata) groups Coarser scales whose unit names to add as columns, e.g. ``["province"]`` Returns ------- geopandas.GeoDataFrame gdf : GeoDataFrame with ``geo_id`` and ``geometry`` (EPSG:4326) df : the panel with ``geo_id``, ``name``, ``year``, the group names and the variables df_dict : one row per ``df`` column with ``var_name``, ``var_def``, ``label``, ``type`` (entity / time / factor / numeric), ``role`` and ``can_be_na`` Example: ```python !pip install -q geo4bolivia geometrics import geo4bolivia as geo import geometrics as gm gdf, df, df_dict = geo.for_geometrics("sdg", variables="index_sdg") df = gm.set_labels(df, df_dict, set_panel=True) w = gm.make_weights(gdf, method="queen") gm.explore_moran_plot(df, "index_sdg", gdf=gdf, w=w, period=2024).fig ```