The Global Liveability Survey’s less than compelling methods, and an alternative

by Chris Auld

The Economist’s liveability Scale is under fire for downgrading Vancouver in the latest results. The reason given for the downgrade: traffic problems on the Malahat, a nearby mountain. The problem: the Malahat is on Vancouver Island, two hours away by ferry. How does the Economist calculate their scale, and are there better alterntatives?

How did traffic problems on the Malahat reflect in the scale’s rankings so quickly? While traffic through the Malahat has long been an issue, the recent widely-reported problem was a twenty-two hour closure this April.

Most attempts to quantify living standards are based on weighted averages of various objective measures. The Human Development Index, for example, is based on measures of life expectancy, education, and income. If the Economist’s livability scale were methodologically similar, then it would have to both be the case that the Malahat is included in the Vancouver region and that problems on the Malahat had already somehow appeared in data, which seems unlikely.

It turns out the liveability is based largely on the completely subjective impressions of the people who write the report.

The Economist Intelligence Unit’s liveability rating quantifies the challenges that might be presented to an individual’s lifestyle in any given location, and allows for direct comparison between locations. Every city is assigned a rating of relative comfort for over 30 qualitative and quantitative factors across five broad categories: stability; healthcare; culture and environment; education; and infrastructure. Each factor in a city is rated as acceptable, tolerable, uncomfortable, undesirable or intolerable. For qualitative indicators, a rating is awarded based on the judgment of in-house analysts and in-city contributors. For quantitative indicators, a rating is calculated based on the relative performance of a number of external data points. The scores are then compiled and weighted to provide a score of 1–100, where 1 is considered intolerable and 100 is considered ideal.

It would appear, then, that traffic is one of the qualitative scales, analysts create these scales at least in part by reading news reports, and they incorrectly assumed reports of problems on Vancouver Island pertain to Vancouver.

This is terrible methodology even putting aside the geographic illiteracy. Why choose these 30 categories and not 30, or 60, three, other categories? Given the categories chosen, why choose a five point scale—the answers would probably change, everything else equal, with a six or 12 or 100 point scale. The individual measures are not quantitative except in that they are an ordinal ranking: there is no meaningful sense in which a one point change from “intolerable” to “undesirable” is the same as a one point change from “tolerable” to “acceptable,” for example. Adding up the individual scales, even with weights, is then meaningless. The measures are weighted, presumably to reflect issues like severe crime causing lower livability than severe seagull droppings, but the weights used cannot be anything other than subjective and arbitrary. So we wind up with an arbitrarily weighted sum of arbitrary measures which can neither be meaningfully summed nor measured. That’ll be $5,250 for the latest issue, by the way.

There are more rigorous alternatives. The distribution of land prices and wages across cities in part reflects differences in amenities across cities. Other things equal, you will have to pay more for a unit of land in a city which is a nice place to live than you will for a piece of land in a dump. Looking through the lens of a model which sorts out how wages and rents are jointly determined across cities, we can statistically infer quality of life from objective data on prices and a few other characteristics. A recent example is Albouy 2010. Here is a graph from that paper showing estimates of productivity-enhancing characteristics against quality of life enhancing characterisitcs—essentially, a “livability scale,” formally defined and objectively measured:

Cities with high values on the y-axis have high quality of life, cities with high values on the x-axis have high productivity. The estimates suggest cities like Honolulu, Santa Barbara, and San Francisco have excellent quality of life, which seems plausible. I don’t know why Kokomo, Indiana is last. Perhaps it’s too close to the Malahat.