Exploring the data

A question-driven exploration of multidimensional poverty in Bolivia’s municipalities, and the pitfalls of these data that a newcomer should know.

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab

The Explore page tours the exploration functions one by one. This article uses them the way an analysis does: it starts from questions about one indicator, multidimensional poverty, and picks the function that answers each. Along the way it points out six pitfalls of these data, which are collected in a table at the end.

The questions:

  1. What exactly is the variable?
  2. Where is poverty highest and lowest?
  3. How did its distribution shift between the two periods?
  4. How does it differ by department?
  5. What is it associated with?
  6. What does the trend of night-time lights add?
!pip install -q geo4bolivia
import textwrap

import numpy as np
import geo4bolivia as geo

1. What exactly is the variable?

Exploring a variable you have not read about is how wrong conclusions start. geo.search() finds the candidates:

geo.search("multidimensional poverty")[["dataset", "variable", "label", "years"]]
dataset variable label years
0 sdg01 sdg1_2_mpi Multidimensional poverty index (0-1) 2012, 2024
1 sdg05 sdg5_1_pgm Gender parity in multidimensional poverty (mal... 2012, 2024
2 sdg10 sdg10_2_pel Parity of multidimensional poverty by ethnicit... 2012, 2024
3 sdg10 sdg10_2_mre Parity of multidimensional poverty by migratio... 2012, 2024
4 sdg10 sdg10_2_tpm Parity of multidimensional poverty for older p... 2012, 2024
5 sdg10 sdg10_2_gip Gini coefficient of multidimensional poverty (... 2012, 2024
6 sdg01 index_sdg1 No poverty index (0-100) 2012, 2024
7 sdg05 index_sdg5 Gender equality index (0-100) 2012, 2024
8 sdg10 index_sdg10 Reduced inequalities index (0-100) 2012, 2024

The first one, sdg1_2_mpi in the dataset sdg01, is the index itself. geo.dictionary() documents every variable of the dataset: its label, how it was built, its unit, how it aggregates to coarser scales and the years it really refers to.

d = geo.dictionary("sdg01")
d[["variable", "label", "aggregation", "reference_years"]].iloc[3:]
variable label aggregation reference_years
3 index_sdg1 No poverty index (0-100) weighted_mean:population 2012: ~2012; 2024: ~2024
4 sdg1_2_mpi Multidimensional poverty index (0-1) weighted_mean:population 2012: 2012; 2024: 2024
5 sdg1_4_eep Energy poverty (% of dwellings) weighted_mean:population 2012: 2016; 2024: 2024
6 sdg1_4_nbi Unsatisfied basic needs (% of population) weighted_mean:population 2012: 2012; 2024: 2024
7 sdg1_4_ssb Access to the 3 basic services (% of households) weighted_mean:population 2012: 2012; 2024: 2024
text = d.set_index("variable").loc["sdg1_2_mpi", "description"]
print(textwrap.fill(text.split(" Source:")[0], 84))
Multidimensional poverty index MPI = A × H, where H is the share of households (in
private dwellings) deprived in 3 or more of 9 indicators and A the average intensity
(share of the 9 deprivations) among them. Multidimensional poverty follows Andersen
& Branisa (2020): nine deprivations in three dimensions (resources: drinking water,
electricity, sanitation; power and voice: an illiterate member aged 15+, a member
aged 6+ without an identity card, no fixed or mobile phone; opportunities and
choice: a birth in the last five years not attended by skilled personnel, a teenage
pregnancy in the last five years, a member aged 6-19 not attending school).

So the index runs from 0 to 1, higher means poorer, and it combines how many households are deprived (H) with how deprived they are (A). The rest of the description (cut here at its source) says that it comes from the censuses of 2012 and 2024.

WarningPitfall 1: the two periods are nominal years

The SDG Atlas 2026 describes two periods, “around 2012” and “around 2024”, and every one of its datasets stores them as the years 2012 and 2024. The actual years differ by indicator. For multidimensional poverty they are the census years. For energy poverty (sdg1_4_eep), in the same dataset, the first period is 2016, so its “2012 to 2024” change covers eight years, not twelve. Check geo.reference_years() before computing a rate of change or comparing the changes of two indicators.

geo.reference_years("sdg01", "sdg1_4_eep")
variable label year reference
0 sdg1_4_eep Energy poverty (% of dwellings) 2012 2016
1 sdg1_4_eep Energy poverty (% of dwellings) 2024 2024

2. Where is poverty highest and lowest?

mpi = geo.load("sdg01", variables="sdg1_2_mpi", groups=["department"])
geo.extremes(mpi, "sdg1_2_mpi", year=2024, n=5)

Multidimensional poverty index (0-1) in 2024 ranges from 0.003 (Nuestra Señora de La Paz) to 0.296 (TIOC-Raqaypampa) across the 343 units with data; the median unit has 0.053.

The 5 highest: TIOC-Raqaypampa (0.296), Poroma (0.223), San Pedro (0.213), El Choro (0.203), Vila Vila (0.2).

The 5 lowest: Nuestra Señora de La Paz (0.003), Colcapirhua (0.005), Trigal (0.006), Santa Cruz de la Sierra (0.006), El Alto (0.006).

The gap between the two ends is 0.293. Extreme values deserve a second look before any analysis: they may be the most interesting units, very small units whose rates are noisy, or errors in the data.

These are associations, not causal effects. A causal reading needs a research design.

In 2024 the index ranges from 0.003 in Nuestra Señora de La Paz to 0.296 in TIOC-Raqaypampa, and the median municipality has 0.053. The lowest values are in the largest cities (La Paz, El Alto, Santa Cruz de la Sierra) and two smaller municipalities, Colcapirhua and Trigal. The highest are spread over four departments: Cochabamba, Chuquisaca, Potosí and Oruro. TIOC-Raqaypampa was also the highest in 2012 (0.563; year=2012 shows that ranking).

A ranking says nothing about where these municipalities are. The map does, and with the two periods in the same classes it also shows the size of the change:

geo.map(mpi, "sdg1_2_mpi", year=[2012, 2024], scheme="fisher_jenks")

The lowest class (up to 0.084) held 20 municipalities in 2012 and holds 261 in 2024. The two highest classes (above 0.273) held 113 in 2012 and hold one in 2024.

3. How did the distribution shift?

geo.distribution(mpi, "sdg1_2_mpi")

Between 2012 and 2024, the mean of Multidimensional poverty index (0-1) fell from 0.234 to 0.0633, and the median went from 0.227 to 0.053.

The spread across units narrowed: the standard deviation went from 0.106 to 0.0441, and the interquartile range (the middle half of the units) from 0.153 to 0.053.

Multidimensional poverty index (0-1) in 2024 averages 0.0633 across 343 units (median 0.053, standard deviation 0.0441). Half of the units lie between 0.03 and 0.083, and the full range runs from 0.003 to 0.296. The distribution is strongly right-skewed: a tail of high values pulls the mean above the median (skewness 1.44).

These are associations, not causal effects. A causal reading needs a research design.

The mean fell from 0.234 to 0.063 and the whole distribution moved towards zero. The dictionary gives two reference points: the Atlas counts the target as met at 0.066 and sees major challenges from 0.23 on. In 2012, 12 municipalities were at or below 0.066 and 164 at or above 0.23; in 2024 the numbers are 221 and 1.

The shape changed as well. In 2012 the distribution was fairly symmetric (skewness 0.46). In 2024 it is squeezed against its floor of zero with a tail of high values (skewness 1.44). Did the gaps between municipalities narrow? It depends on how you measure them, and geo.change(), which gives one row per municipality, shows why:

chg = geo.change(mpi, "sdg1_2_mpi")
cols = ["sdg1_2_mpi_2012", "sdg1_2_mpi_change", "sdg1_2_mpi_pct_change"]
print((chg["sdg1_2_mpi_change"] < 0).sum(), "of", len(chg), "municipalities fell")
chg[cols].corr().round(2)
343 of 343 municipalities fell
sdg1_2_mpi_2012 sdg1_2_mpi_change sdg1_2_mpi_pct_change
sdg1_2_mpi_2012 1.00 -0.95 0.38
sdg1_2_mpi_change -0.95 1.00 -0.10
sdg1_2_mpi_pct_change 0.38 -0.10 1.00

Poverty fell in all 343 municipalities. In absolute terms it fell most where it was highest: the correlation between the 2012 level and the change is −0.95. In relative terms it was the other way round: the percentage fall (76% in the median municipality) was somewhat smaller where poverty was higher (the correlation of the level with the percentage change is +0.38, and a less negative change is a smaller fall).

WarningPitfall 2: skewed variables, and what logs do

For a variable that is bounded at zero and skewed, absolute and relative gaps can tell opposite stories. The standard deviation of the index fell from 0.106 to 0.044: absolute gaps narrowed. The standard deviation of its logarithm, which measures relative gaps, rose:

mpi["ln_mpi"] = np.log(mpi["sdg1_2_mpi"])
geo.distribution(mpi, "ln_mpi").df[["year", "mean", "sd", "skewness"]].round(2)
year mean sd skewness
0 2012 -1.57 0.53 -1.01
1 2024 -3.01 0.75 -0.53

It went from 0.53 to 0.75. Municipalities are closer to each other in points of the index and further apart in proportion. Neither reading is wrong; say which one you use. Variables that are already logs (night-time lights, below) describe relative differences from the start.

4. How does it differ by department?

geo.distribution(mpi, "sdg1_2_mpi", by="department", kind="box")

By Department in 2024 (9 groups), the median of Multidimensional poverty index (0-1) is highest in Potosí (0.0775) and lowest in Tarija (0.023).

Units differ most within Cochabamba (standard deviation 0.0565) and least within Tarija (0.0125). Where the groups’ distributions barely overlap, the differences between groups are large compared with those within them.

In each box, the line is the median and the box holds the middle half of the units; points beyond the whiskers are unusually high or low units.

These are associations, not causal effects. A causal reading needs a research design.

In 2024 the median municipality is poorest in Potosí (0.078) and least poor in Tarija (0.023) and Santa Cruz (0.027). The spread within departments is large: Cochabamba holds both the second lowest municipality of the country (Colcapirhua, 0.005) and the highest (TIOC-Raqaypampa, 0.296).

This figure compares municipalities, each counting the same. It does not give the poverty of a department’s population.

WarningPitfall 3: the mean of the municipalities is not the department’s value

A department’s value has to weight each municipality by its people. geo.aggregate() does that with the rule each variable declares in the metadata: for the SDG indicators, a mean weighted by census population (weighted_mean:population; the weights are loaded from the dataset sdg). The results approximate the department values; they are not the Atlas’s own department figures.

dep = geo.aggregate(mpi, to="department", dataset="sdg01", variables="sdg1_2_mpi")
dep = geo.add_names(dep)
simple = mpi.groupby(["department", "year"], as_index=False)["sdg1_2_mpi"].mean()
dep = dep.merge(simple, left_on=["name", "year"], right_on=["department", "year"],
                suffixes=("_weighted", "_simple"))
dep.loc[dep["year"] == 2024,
        ["name", "sdg1_2_mpi_weighted", "sdg1_2_mpi_simple", "n_units"]].round(3)
name sdg1_2_mpi_weighted sdg1_2_mpi_simple n_units
1 Chuquisaca 0.058 0.090 29
3 Cochabamba 0.038 0.070 48
5 Beni 0.072 0.075 20
7 La Paz 0.027 0.060 87
9 Oruro 0.037 0.069 35
11 Pando 0.049 0.068 15
13 Potosí 0.063 0.083 42
15 Santa Cruz 0.017 0.032 56
17 Tarija 0.018 0.027 11

The two columns differ by a factor of about two in several departments. In La Paz, the simple mean of the 87 municipalities is 0.060 and the population-weighted value 0.027, because more than half of the department’s people live in two municipalities with very low poverty (El Alto and Nuestra Señora de La Paz). The same holds for the country: aggregating with to="country" gives 0.133 in 2012 and 0.033 in 2024, against municipal means of 0.234 and 0.063. Every “mean” in the exploration functions is a mean across municipalities.

5. What is it associated with?

Start inside the dataset. With a dataset id, geo.correlation() uses every variable of it:

geo.correlation("sdg01", year=2024)

Pearson correlations between 5 variables in 2024, on the 343 observations with data for all of them.

The strongest positive correlation is between Multidimensional poverty index (0-1) and Unsatisfied basic needs (% of population) (r = 0.82): a strong association, in which units with more of one tend to have more of the other.

The strongest negative correlation is between No poverty index (0-100) and Unsatisfied basic needs (% of population) (r = -0.94): a strong association, in which units with more of one tend to have less of the other.

The weakest correlation, in either direction, is between Energy poverty (% of dwellings) and Access to the 3 basic services (% of households) (r = -0.46).

A coefficient near +1 or -1 means two variables move together closely, in the same or in opposite directions; near 0, that knowing one says little about the other. The table (.gt) sets Pearson correlations (above the diagonal) against Spearman rank correlations (below): a large gap between the two points to a non-linear relation or to outliers.

These are associations, not causal effects. A causal reading needs a research design.

Multidimensional poverty moves closely with unsatisfied basic needs (r = 0.82), INE’s own measure of deprivation; less closely with energy poverty (0.54); and negatively with access to the three basic services (−0.69). Its correlation with the goal index index_sdg1 (−0.82) is not a finding: the index is built from these four indicators, so it is correlated with them by construction.

Other candidates live in other datasets. geo.combine() puts several datasets on one scale, as one panel; a dictionary picks the variables.

both = geo.combine({"sdg01": ["sdg1_2_mpi"],
                    "sdg": ["index_sdg", "urbanization", "population"]},
                   scale="municipality")
both = geo.add_names(both, groups=["department"])
variables = ["sdg1_2_mpi", "index_sdg", "urbanization", "population"]
res = geo.correlation(both, variables)
res.gt
Pearson correlations, 2024
A B C D
A: Multidimensional poverty index (0-1) -0.73 -0.46 -0.20
B: Sustainable Development Index, IDS (0-100) -0.79 0.56 0.37
C: Urbanization rate (% of population) -0.49 0.50 0.39
D: Population (census) -0.24 0.28 0.58
Pearson correlations above the diagonal, Spearman rank correlations below. Observations: 343.

The table shows Pearson coefficients above the diagonal and Spearman (rank) coefficients below. Poverty is lower where the Sustainable Development Index is higher (−0.73; again partly by construction, since the index averages the goal indices and the poverty goal is one of them), where the urbanization rate is higher (−0.46) and, weakly, where the population is larger (−0.20).

WarningPitfall 4: correlations use complete cases

geo.correlation() keeps only the rows with data for every variable, so that all coefficients refer to the same units. One variable with gaps therefore removes rows from all the pairs. urbanization exists only for 2024, so pooling both periods with year="all" uses half of the panel, and the result says so in .notes:

pooled = geo.correlation(both, variables, year="all")
print(pooled.n, "observations |", pooled.notes[0])
343 observations | 343 of 686 observations lack data for at least one variable and were left out.

Population is the kind of variable for which a straight line on the raw scale is a poor summary: the largest city has about 130 times the population of the median municipality. log_x=True puts the horizontal axis in logarithms.

geo.scatter(both, "population", "sdg1_2_mpi", log_x=True, color="department")

Across 343 units in 2024, Multidimensional poverty index (0-1) and Population (census) are negatively correlated (r = -0.23, a weak association): units with higher Population (census) tend to have lower Multidimensional poverty index (0-1).

The least-squares line has a slope of -0.00903: 1% more of Population (census) goes with 9.03e-05 lower Multidimensional poverty index (0-1), on average (Population (census) is in logs). The line accounts for 5% of the variation in Multidimensional poverty index (0-1) across units (R² = 0.05); the slope is statistically significant at the 1% level, by a test that treats the units as independent of each other.

These are associations, not causal effects. A causal reading needs a research design.

Even in logs the association is weak (r = −0.23): the line accounts for 5% of the variation in poverty. The 17 municipalities with more than 100,000 people mostly have very low poverty (their median is 0.014), while the 55 with fewer than 5,000 range from 0.006 to 0.177. These are associations between municipal averages in one year, and they say nothing about why some municipalities are poorer.

6. What does the trend of night-time lights add?

Poverty is observed twice, twelve years apart. Night-time lights, measured by satellite, are observed every year from 2012 to 2020; the dataset’s description calls them a widely used proxy for local economic activity and electrification. The variable ln_NTLpc is the logarithm of radiance per person.

ntl = geo.load("ntl", groups=["department"])
res = geo.trend(ntl, "ln_NTLpc")
res

Between 2012 and 2020, the mean of Log night-time lights per capita across 331 units rose from 13.1 to 13.9 (a change of +0.812).

94% of the units with data in both years ended higher than they started: the summary line describes the typical unit, not every unit.

The band is one standard error on each side of the mean: how precisely the average is known, not how much units differ.

These are associations, not causal effects. A causal reading needs a research design.

The mean of the logarithm rose from 13.1 to 13.9. A difference of 0.81 in a logarithm is a multiplication by about 2.25 (exp(0.81)), and 94% of the municipalities with data in both years ended higher. Most of the rise came in the first years: the mean reached 13.9 in 2017 and has stayed near that level since. That is what an annual series adds to two census snapshots: the path between them.

Two more pitfalls come with this dataset.

WarningPitfall 5: 333 units, not 343, and gaps within them

pop, ntl and the satellite embeddings were computed on the municipal map of 2015. They are published for the 333 municipalities whose territory has not changed since, and not for the four units created since then or the six that lost territory to them. On top of that, ln_NTLpc is missing in the years in which no lit pixel remained in a municipality.

print(res.notes[0])
units = geo.units("municipality")
missing = units.loc[~units["geo_id"].isin(ntl["geo_id"]), ["geo_id"]]
missing.merge(mpi[mpi["year"] == 2024]).sort_values("sdg1_2_mpi")[
    ["geo_id", "name", "department", "sdg1_2_mpi"]]
The number of units with data changes over time (from 318 to 331), so part of the change may come from which units are observed.
geo_id name department sdg1_2_mpi
5 BO-07-02-01 Porco Potosí 0.024
0 BO-02-12-02 Mizque Cochabamba 0.068
6 BO-07-02-02 Tomave Potosí 0.079
2 BO-03-06-01 San Ignacio Beni 0.084
7 BO-07-02-04 TIOC-Jatun Ayllu Yura Potosí 0.089
3 BO-03-08-02 Santa Ana Beni 0.098
9 BO-07-04-05 San Pedro de Macha Potosí 0.123
8 BO-07-04-01 Colquechaca Potosí 0.136
4 BO-03-09-01 TIOC-Territorio Indígena Multiétnico Beni 0.168
1 BO-02-12-04 TIOC-Raqaypampa Cochabamba 0.296

The ten units left out are not a random sample: they include TIOC-Raqaypampa, the poorest municipality in both periods. And because the number of municipalities with data grows from 318 in 2012 to 331 in 2019, part of the trend comes from dimly lit municipalities entering the sample. The dataset has a second variable for this, ln_t400NTLpc, a smoothed trend that is complete for all 333 units in every year; on it, the mean rises by 1.01 instead of 0.81.

How do lights relate to poverty? The two datasets share one year, 2012, and geo.scatter() uses it when asked:

lights = geo.combine({"sdg01": ["sdg1_2_mpi"], "ntl": ["ln_NTLpc"]},
                     scale="municipality")
geo.scatter(lights, "ln_NTLpc", "sdg1_2_mpi", year=2012)

Across 318 units in 2012, Multidimensional poverty index (0-1) and Log night-time lights per capita are negatively correlated (r = -0.58, a moderate association): units with higher Log night-time lights per capita tend to have lower Multidimensional poverty index (0-1).

The least-squares line has a slope of -0.0427: one more unit of Log night-time lights per capita goes with 0.0427 lower Multidimensional poverty index (0-1), on average. The line accounts for 34% of the variation in Multidimensional poverty index (0-1) across units (R² = 0.34); the slope is statistically significant at the 1% level, by a test that treats the units as independent of each other.

These are associations, not causal effects. A causal reading needs a research design.

Across the 318 municipalities with both values, poverty is lower where lights per person are higher (r = −0.58): one more unit of the logarithm (2.7 times the radiance per person) goes with an index 0.043 lower, on average. Lights account for about a third of the variation in poverty (R² = 0.34), so they are an informative but rough proxy.

WarningPitfall 6: not every variable can be aggregated

A mean of logarithms is not the logarithm of the department’s lights per person, so the metadata gives ln_NTLpc the rule none and geo.aggregate() refuses to guess:

try:
    geo.aggregate(ntl, to="department", dataset="ntl", variables="ln_NTLpc")
except ValueError as error:
    print(error)
No aggregation rule for: ln_NTLpc. Pass rules={'ln_NTLpc': 'mean'} (one of sum, mean, min, max, weighted_mean:<field>, none) or leave these variables out with variables=[...].

To compare departments, either ask for the mean of the municipalities explicitly (rules={"ln_NTLpc": "mean"}) and describe it as such, or rebuild the totals (lights = exp(ln_NTLpc) × population), sum them and take the logarithm again, as Building your own indicators does. The result’s n_units and coverage columns tell how many municipalities each department value rests on.

The pitfalls, in one table

Pitfall Where it shows What to do
Two periods stored at nominal years every sdg dataset geo.reference_years() before computing rates or comparing changes
Skewed variables poverty near zero, population look at the distribution; use logs or ranks for relative differences, and say which
Municipal means are not department values any by="department" summary geo.aggregate() with the metadata rules; they approximate the official figures
Complete cases geo.correlation() with a variable that has gaps read .n and .notes; correlate fewer variables or one year
333 versus 343 units pop, ntl, emb, embpw check which units are missing; they are not a random sample
Variables without an aggregation rule logs, trends, ranks aggregate the underlying totals, or state an explicit rule

In short

  • Multidimensional poverty fell in all 343 municipalities between the censuses of 2012 and 2024; the municipal mean went from 0.234 to 0.063.
  • Absolute gaps between municipalities narrowed and relative gaps widened: the distribution is now skewed, with a tail of high values.
  • Departments differ, and municipalities within a department differ more. Population-weighted department values are about half the simple municipal means in several departments.
  • Poverty is associated with unsatisfied basic needs, lower urbanization and lower night-time lights per person. None of these associations says what causes what.
  • Read the metadata first: the reference years, the units covered and the aggregation rule decide what a number means.

Where next