Transitions and mobility

How units move inside a distribution: changes of rank between two years, moves between spatial cluster types, and Markov transition matrices with and without the neighbours.

✨ Run this page yourself: nothing to install on your computer.

▶ Open in Colab

A mean, a Gini index or a convergence slope describe a distribution as a whole. They do not say who moved. Two years can have the same dispersion with every unit in the same place, or with the top and the bottom having swapped. This article is about the movement of units inside the distribution, with four functions:

Question Function Years needed
How much did the ranking change, and who climbed or fell? geo.rank_mobility 2
Did the spatial clusters stay where they were? geo.lisa_transitions 2
How likely is a unit to move from one class of the distribution to another in a year? geo.transitions 2, better many
Do those chances depend on the unit’s neighbours? geo.spatial_transitions 2, better many

The first two compare two years, and run here on the two periods of the Sustainable Development Index. The last two follow units year by year, and run on nine years of night-time lights. They use the giddy library, which the [dynamics] extra installs.

!pip install -q "geo4bolivia[dynamics]"
import pandas as pd
import geo4bolivia as geo

ids = geo.load("sdg", variables="index_sdg", groups=["department"])

Rank mobility: who climbed and who fell?

geo.rank_mobility() ranks the units in each of two years (rank 1 is the highest value) and compares the two rankings.

rm = geo.rank_mobility(ids, "index_sdg")
rm

The 343 units with data in 2012 and 2024 were ranked by Sustainable Development Index, IDS (0-100) in each year (rank 1 = the highest value). The Spearman correlation of the two rankings is 0.843 and Kendall’s tau is 0.666: the ordering is broadly preserved, with visible reshuffling. Of all pairs of units, about 83% are in the same order in both years.

A unit moved 39.8 positions on average (median 29), 12% of the length of the ranking; two unrelated rankings would differ by about 114 positions. 163 units climbed, 177 fell and 3 kept their rank.

56% of the units are in the same quantile group in both years, out of 5 groups (about 20% would be if the two years were unrelated). The group that kept most of its units is Q5 (77% stayed) and the one that kept fewest is Q3 (37%); Q1 holds the lowest values and Q5 the highest.

The largest climb is Cruz de Machacamarca, from rank 315 to 90; the largest fall is San José de Chiquitos, from rank 42 to 283.

Ranks are relative: a unit can fall while its value rises, when the others rose more. Where values are close together, often in the middle of the distribution, small differences move a unit many positions.

These are associations, not causal effects. A causal reading needs a research design.

Each point is a municipality: its rank in 2012 against its rank in 2024. Points on the dashed line kept their position; the further from the line, the larger the move.

Two rank correlations summarise how much of the order survived:

print("Spearman:", round(rm.spearman, 3), " Kendall:", round(rm.kendall, 3))
print("positions moved, mean:", round(rm.mean_abs_change, 1),
      " median:", rm.median_abs_change)
print(rm.n_climbed, "climbed,", rm.n_fell, "fell, of", rm.n_units)
Spearman: 0.843  Kendall: 0.666
positions moved, mean: 39.8  median: 29.0
163 climbed, 177 fell, of 343
  • Spearman’s correlation is the ordinary correlation computed on the ranks: 1 if the order is identical, 0 if the two rankings are unrelated. Here it is 0.84.
  • Kendall’s tau counts pairs of units. It is the share of pairs in the same order in both years minus the share that swapped: 0.67, which means about 83% of all pairs of municipalities kept their order.
  • A municipality moved 40 positions on average (the median is 29) in a ranking of 343. Three kept their rank: Nuestra Señora de La Paz, Tarija and Santa Cruz de la Sierra, at positions 1, 2 and 6.

So the order was broadly kept, with a good deal of reshuffling. rm.gt lists the units that moved most (top= sets how many), and rm.climbers and rm.fallers hold them as tables:

rm.gt
Rank mobility: Sustainable Development Index, IDS (0-100)
2012 to 2024; 343 units; rank 1 = the highest value; Spearman correlation 0.84
Unit Rank 2012 Rank 2024 Change Value 2012 Value 2024
Climbed the most
Cruz de Machacamarca 315 90 +225 42.90 63.40
Carangas 304 100 +204 43.40 62.80
Sicaya 318 153 +165 42.80 60.40
Todos Santos 290 135 +155 44.40 61.20
Villa Gualberto Villarroel 304 156 +148 43.40 60.30
Bolpebra 250 125 +125 46.20 61.70
Santa Rosa 202 83 +119 48.20 63.80
Escara 293 179 +114 44.20 59.30
Loreto 304 194 +110 43.40 58.60
Laja 232 123 +109 47.10 61.80
Fell the most
San José de Chiquitos 42 283 −241 57.80 55.10
San Antonio de Esmoruco 168 316 −148 49.70 53.20
San Miguel de Velasco 149 286 −137 50.40 54.90
Puerto Suárez 52 188 −136 56.50 58.80
Roboré 41 175 −134 58.30 59.50
Cabezas 106 239 −133 52.40 57.30
Totora 186 314 −128 48.90 53.40
Santa Ana 96 224 −128 52.90 57.70
Urubichá 177 302 −125 49.10 54.20
Ascención de Guarayos 215 337 −122 47.60 50.20

The largest climb is Cruz de Machacamarca, from rank 315 to 90, with an index that rose from 42.9 to 63.4. The largest fall is San José de Chiquitos, from 42 to 283: it is the one municipality whose index went down (57.8 to 55.1). The other units in the lower half of the table all improved their index, by less than the rest. A fall in rank is not a fall in the value.

The quantile transition matrix

The second view sorts the units into k groups of equal size in each year (quintiles by default; Q1 has the lowest values) and counts where the members of each group ended up.

rm.fig_matrix

Each row is a group of 2012 and sums to one. The diagonal is the share that stayed: 56% of all municipalities are in the same quintile in both years (rm.share_same_group), against the 20% expected if the two years were unrelated. The ends hold their members best: 77% of the top quintile and 67% of the bottom one stayed, against 37% of the middle quintile. That is partly arithmetic: a unit at the top can only move down, and in the middle of a distribution values are close together, so a small change in the index crosses many positions.

What rank mobility does not show:

  • Levels. Ranks are relative. The index rose in 342 of the 343 municipalities, and half of them still fell in the ranking, because a ranking has as many moves down as up.
  • Distances. A move of 50 positions in the crowded middle can be a smaller change of the index than a move of 5 positions at the top.
  • Noise. With two periods, a unit whose first value was unusually low or high will move towards its usual place, and that looks like mobility.

For variables where less is better (poverty, mortality), ascending=True makes rank 1 the lowest value.

LISA transitions: did the clusters stay in place?

Local Moran statistics (LISA) give each unit a type according to its value and its neighbours’ values, both compared with the mean: High-High (a high value among high neighbours, a hot spot), Low-Low (a cold spot), High-Low and Low-High (outliers that differ from their surroundings), or Not significant when the pattern is compatible with chance. The spatial dependence article explains them.

geo.lisa_transitions() computes the types in two years, on the same units and the same weights, and counts the moves between them.

lt = geo.lisa_transitions(ids, "index_sdg")
lt

Local Moran statistics (LISA) of Sustainable Development Index, IDS (0-100) were computed in 2012 and in 2024 for the same 343 units, under queen contiguity (1 island(s) attached to nearest neighbor), row-standardized, n=343, and each unit was assigned to a type at the 0.05 level. Global Moran’s I is 0.432 in 2012 (pseudo p = 0.001) and 0.468 in 2024 (pseudo p = 0.001): the overall clustering of similar values changed little.

275 of 343 units (80%) are of the same type in both years and 68 changed type. Of the 104 units in a significant cluster or outlier in 2012, 71 kept their type, 23 are no longer significant in 2024 and 10 switched to another type. 35 units that were not significant in 2012 belong to a cluster or are an outlier in 2024.

Of the 46 High-High hot spots (high values surrounded by high values) of 2012, 39 are still High-High in 2024, a year with 60 of them. Of the 44 Low-Low cold spots (low values surrounded by low values) of 2012, 30 are still Low-Low in 2024, a year with 45 of them. The most common change is from Not significant to High-High (17 units).

Read the changes with care. A unit’s type depends on whether its pseudo p-value is below the threshold, computed unit by unit without a multiple-testing adjustment, so moves to and from Not significant can come from borderline p-values. High and low are relative to the mean of each year, and a unit can change type because its neighbours’ values changed while its own did not.

These are associations, not causal effects. A causal reading needs a research design.

Rows are the types in 2012, columns the types in 2024, and each cell counts municipalities.

  • Most units kept their type: 275 of 343 (80%). Most of those, 204, are not significant in either year.
  • Hot spots persisted and grew in number. Of the 46 High-High municipalities of 2012, 39 are still High-High in 2024, and 17 that were not significant joined them: 60 in 2024.
  • Cold spots persisted less. Of the 44 Low-Low municipalities of 2012, 30 are still Low-Low; 13 are no longer significant, and 13 others became Low-Low: 45 in 2024.
  • Nothing crossed over. No municipality went from High-High to Low-Low or the other way.

Global Moran’s I, the overall tendency of similar values to be neighbours, changed little: 0.43 in 2012 and 0.47 in 2024 (lt.moran_i_first, lt.moran_i_last). The moves in words, from the most common:

lt.df[lt.df["moved"]]["transition"].value_counts().head(6)
transition
Not significant → High-High    17
Low-Low → Not significant      13
Not significant → Low-Low      13
Low-High → High-High            4
High-High → Not significant     4
High-Low → Not significant      3
Name: count, dtype: int64

The map shows the types of 2024, with the municipalities that changed type outlined:

lt.fig_map

Read the changes with care:

  • The threshold. A unit has a type only when its pseudo p-value is below 0.05 (alpha=). Units near the threshold move in and out of “Not significant” without much having changed, and 58 of the 68 moves here are to or from that category.
  • Relative to each year’s mean. “High” and “low” are measured against the mean of the same year. The index rose almost everywhere, so a Low-Low municipality of 2024 usually has a higher index than it had in 2012.
  • The neighbours count. A unit can change type because its neighbours’ values changed while its own did not.

Markov transitions: how likely is a move?

With many years, the same counting can be done for every pair of consecutive years and turned into probabilities. geo.transitions() assigns each unit to one of k classes in every year and builds the transition matrix: for a unit in a given class this year, the probability of being in each class next year.

The data are night-time lights per capita (ntl, the variable ln_NTLpc), yearly from 2012 to 2020.

tr = geo.transitions("ntl", "ln_NTLpc", k=5)
tr

Pooling 2,528 year-to-year moves, Log night-time lights per capita was discretized into 5 quantile states and its movement summarized by a transition-probability matrix: each row gives the chance of ending the next year in each state, conditional on the current one. On average a unit stays in its current state with probability 0.83 — the diagonal is dominant, so positions in the distribution are highly persistent. The stickiest state is Q5 (stay probability 0.909); the most mobile is Q2 (0.758). The Shorrocks mobility index is 0.212 on a 0 (complete immobility: the identity matrix) to 1.25 (all mass off the diagonal) scale; the Prais determinant index is 0.639 and the Bartholomew index 0.212. If these transition probabilities kept operating, the cross-section would settle into the ergodic (steady-state) mix with the largest long-run share in Q1 (0.203, versus 0.2 under an even split). Expected sojourn times say a unit entering Q5 remains there for about 11 consecutive years before moving.

These are associations, not causal effects. A causal reading needs a research design.

print(tr.notes)
tr.p.round(3)
('transitions: kept the 316 units with data in all 9 years (15 left out)',)
Q1 Q2 Q3 Q4 Q5
Q1 0.898 0.100 0.002 0.000 0.000
Q2 0.099 0.758 0.137 0.004 0.002
Q3 0.004 0.141 0.766 0.087 0.002
Q4 0.000 0.000 0.093 0.819 0.087
Q5 0.000 0.000 0.002 0.089 0.909

The classes are quintiles formed in each year, so they are positions in that year’s distribution: Q1 is the fifth of municipalities with the least light per capita. (The variable is a logarithm; quintiles do not change when a variable is logged, so that makes no difference here.) The matrix pools 2,528 yearly moves of the 316 municipalities observed in all nine years.

  • Positions are persistent from one year to the next. A municipality stays in its class with a probability of 0.83 on average. The ends are the most persistent: 0.90 for Q1 and 0.91 for Q5; in the middle classes it is 0.76 to 0.82.
  • Moves are short. Almost all the probability off the diagonal is in the classes next to it. From Q1, the chance of reaching Q3 in one year is 0.002, and no unit went further.

The table of the result adds three kinds of summary:

tr.gt
Markov transition summary — Log night-time lights per capita
5 states (quantiles classes, per-period classification), 2,528 transitions
Measure Value
Mobility indices
Shorrocks trace index (giddy measure 'P') 0.212
Prais determinant index (giddy measure 'D') 0.639
Bartholomew index (giddy measure 'B1') 0.212
Ergodic (steady-state) distribution
Long-run share in Q1 0.203
Long-run share in Q2 0.199
Long-run share in Q3 0.199
Long-run share in Q4 0.199
Long-run share in Q5 0.199
Expected sojourn time (periods)
Consecutive periods in Q1 9.846
Consecutive periods in Q2 4.131
Consecutive periods in Q3 4.271
Consecutive periods in Q4 5.538
Consecutive periods in Q5 10.957
  • Mobility indices condense the matrix in one number. Shorrocks’ index is 0 when no unit ever moves and 1 when next year’s class does not depend on this year’s: 0.21 here.
  • Expected stay (sojourn time) is the number of consecutive years a unit is expected to remain in a class, 1 / (1 − the probability of staying): about 10 years in Q1, 11 in Q5 and 4 in Q2.
  • The steady state is the distribution the units would settle into if the same matrix applied year after year, for ever.

Here the steady state is 20% in each class, and it could be nothing else: quintiles formed year by year hold 20% of the units every year by construction. To ask where the distribution is heading, the classes have to be fixed levels. per_year=False forms them over all the years together, and relative=True first divides each value by the mean of its year, so that a class is a position relative to the average municipality:

fixed = geo.transitions("ntl", "ln_NTLpc", k=5, per_year=False, relative=True)
fixed.steady_state.round(2)
Q1    0.17
Q2    0.22
Q3    0.25
Q4    0.23
Q5    0.12
Name: steady_state, dtype: float64

With fixed classes the long-run distribution has fewer units at both ends (17% in the lowest class and 12% in the highest, against 20% of the observations in each) and more in the middle. This is the same movement that σ-convergence measures on these data as a falling dispersion, seen through the transition matrix.

The steady state is an extrapolation. It assumes that the probabilities are the same every year (eight yearly steps are little to check that), that a unit’s chances depend only on its present class and not on how it got there, and that the same matrix will hold in the future. Read it as “the direction in which these eight years of moves point”, not as a forecast.

Spatial Markov: do the neighbours matter?

The transition matrix treats all units in a class alike. Rey’s spatial Markov chain asks whether a unit’s chances depend on where its neighbours stand. geo.spatial_transitions() classifies each unit’s neighbourhood (the average of its neighbours, the spatial lag) as well as the unit, and estimates one transition matrix for each class of neighbourhood. If space does not matter, the matrices are equal apart from chance, and two tests check that.

By default the classes are fixed levels relative to each year’s mean. Three classes instead of five keep enough moves in every row of every matrix:

st = geo.spatial_transitions("ntl", "ln_NTLpc", k=3)
st

The spatial Markov chain splits Log night-time lights per capita’s 3-state transition matrix by the neighbors’ position (spatial lag under queen contiguity (1 island(s) attached to nearest neighbor), row-standardized, n=316), giving one matrix per neighborhood class (3 classes); values were expressed relative to each year’s mean first. The unconditional matrix keeps units in place with average probability 0.914. Conditioning on context, units surrounded by low-value neighbors stay in place with average probability 0.903, while units surrounded by high-value neighbors do so with 0.896 — similar persistence in both contexts. The homogeneity tests (LR = 23.1, p = 0.0102; Q = 23.9, p = 0.00791; 10 degrees of freedom) are statistically significant at the 1% level: transition dynamics differ across neighborhood contexts — a unit’s mobility is associated with the state of its neighbors.

These are associations, not causal effects. A causal reading needs a research design.

st.gt
Spatial homogeneity tests — Log night-time lights per capita
H0: transition dynamics identical across the 3 neighbor classes — queen contiguity (1 island(s) attached to nearest neighbor), row-standardized, n=316
Test Statistic dof p-value
Likelihood ratio (LR) 23.138 10 0.0102
Chi-square (Q) 23.887 10 0.0079

Each panel of the figure is the transition matrix of the units whose neighbours are in one class, from the lowest (Q1) to the highest (Q3). Compare the same row across panels:

pd.concat({f"neighbours in Q{i + 1}": p.round(3)
           for i, p in enumerate(st.p_conditional)})
Q1 Q2 Q3
neighbours in Q1 Q1 0.935 0.063 0.002
Q2 0.093 0.872 0.035
Q3 0.000 0.099 0.901
neighbours in Q2 Q1 0.891 0.109 0.000
Q2 0.082 0.886 0.032
Q3 0.000 0.079 0.921
neighbours in Q3 Q1 0.833 0.167 0.000
Q2 0.044 0.898 0.058
Q3 0.000 0.045 0.955
  • A municipality in the lowest class is more likely to leave it when its neighbours are higher. The probability of moving from Q1 to Q2 in a year is 0.06 with neighbours in the lowest class, 0.11 with neighbours in the middle class and 0.17 with neighbours in the highest.
  • A municipality in the highest class is more likely to keep it when its neighbours are high too. The probability of falling from Q3 to Q2 is 0.10, 0.08 and 0.04 across the same three neighbourhoods.
  • The tests reject equal matrices: the likelihood ratio test gives p = 0.010 and the chi-square test p = 0.008, with 10 degrees of freedom.

The average probability of staying is about the same in low and in high neighbourhoods, which is what the reading above reports; the difference is in the direction of the moves. Low neighbours go with staying low, high neighbours with staying high or moving up.

What this does not show:

  • Not a spillover. Neighbouring municipalities can share roads, climate, markets and administrative history. Any of these makes neighbours move together without one affecting the other.
  • Thin cells. The moves are split over classes of units and classes of neighbourhood, and some combinations are rare: municipalities in the lowest class with neighbours in the highest account for 60 of the 2,528 moves. With k=5 the same row rests on 7. Check the counts before reading a probability:
moves = st.df.sort_values(["geo_id", "year"])
moves["next_state"] = moves.groupby("geo_id")["state"].shift(-1)
pd.crosstab([moves["neighbor_state"], moves["state"]], moves["next_state"])
next_state Q1 Q2 Q3
neighbor_state state
Q1 Q1 489 33 1
Q2 21 197 8
Q3 0 10 91
Q2 Q1 237 29 0
Q2 31 334 12
Q3 0 15 176
Q3 Q1 50 10 0
Q2 10 203 13
Q3 0 25 533
  • The units left out. Only municipalities with lights in every year enter (316 of 333), and a unit’s neighbours are those among the 316.

A transition matrix from two periods

geo.transitions() also runs on two years. On the Sustainable Development Index it gives the matrix that rank_mobility showed above:

two = geo.transitions(ids, "index_sdg", k=5)
print(two.n_transitions, "moves, one per municipality")
two.p.round(2)
343 moves, one per municipality
Q1 Q2 Q3 Q4 Q5
Q1 0.67 0.22 0.07 0.04 0.00
Q2 0.21 0.51 0.23 0.04 0.00
Q3 0.10 0.19 0.37 0.27 0.06
Q4 0.00 0.09 0.28 0.46 0.18
Q5 0.01 0.00 0.03 0.19 0.77

What it can say: how the municipalities of each quintile of 2012 are distributed over the quintiles of 2024. That is a description of one observed step, and a useful one.

What it cannot say:

  • Whether the probabilities are stable. There is one step, so there is nothing to compare it with. Each row rests on about 69 municipalities.
  • Anything per year. The step is about twelve years long. An “expected stay of 4 periods” in this matrix would mean 4 steps of twelve years.
  • Where the distribution is heading. A steady state repeats the same step for ever. With one observed step, between two nominal years, that is arithmetic and not evidence.
  • Real moves from noise. A unit near the border of two classes changes class with a small change of its value, or with an error in either year. With many years such flips show up as moves back and forth; with two they cannot be told apart from lasting moves.

With two periods, stay with the description: rank_mobility for who moved and by how much, and lisa_transitions for the clusters.

In short

  • geo.rank_mobility: between ~2012 and ~2024 the ranking of the Sustainable Development Index was broadly kept (Spearman 0.84, Kendall 0.67), a municipality moved 40 positions on average, and 56% stayed in their quintile.
  • geo.lisa_transitions: 80% of the municipalities kept their LISA type; hot spots went from 46 to 60 and cold spots from 44 to 45, and most changes are to or from “Not significant”.
  • geo.transitions: from one year to the next, a municipality stays in its quintile of night-time lights per capita with a probability of 0.83, and moves are almost always to the next class.
  • geo.spatial_transitions: those chances differ with the neighbours’ class (p = 0.01): low neighbours go with staying low, high neighbours with moving up.
  • Classes formed year by year describe positions; a steady state needs fixed classes, and is an extrapolation in any case.
  • A transition matrix from two periods describes one step. It does not support long-run statements.

Where next