New challenges for Consumer Price Indices: a view from the UK

There are a number of considerations for national statistical agencies looking to take maximum advantage of the data revolution, write Kevin Fox, Peter Levell and Martin O’Connell

A quiet revolution is underway in how consumer price data are being collected. For over 60 years, the price data underlying measures of consumer price inflation in the UK such as the Retail Prices Index (RPI) and the Consumer Prices Index (CPI), have been collected in much the same way. The CPIH is the UK leading measure of inflation. 

The Office for National Statistics (ONS) decides on a shopping basket of “representative items” which is updated annually. This year, the basket included around 700 items chosen for their economic importance as well as statistical considerations such as whether goods will be available throughout the year and whether they are representative of broader price movements (this year, men’s suits and doughnuts were taken out of the basket and meat-free sausages and sports bras were added in). 

The prices of these items are then collected in a large price survey, with quotes collected each month from different shop types and regions. A price index – like the CPIH or RPI – calculates how the cost of this basket changes over the course of the year, weighting individual items according to their importance in consumer spending. “Chaining” the price changes of these shopping baskets across years allows us to see how much prices have risen relative to the price level in previous years, when the shopping basket may have been different.

How to produce a meaningful CPI during COVID-19 lockdown

This approach is tried and tested but has a number of limitations. The items need to be chosen such that are available across all months of the year, excluding highly seasonal goods. Manually collecting prices is also laborious and limits the sample of price quotes that can be collected. These shortcomings mean that the CPI is much better for estimating annual price changes (e.g., from March 2021 to March 2022) than for measuring changes from one month to the next (e.g., from February 2022 to March 2022).

This is changing as part of a multi-year development plan for consumer price statistics. As part of this, the ONS will source price quotes directly from supermarkets without the need for a survey. Many of these retailers are already sending the ONS regular feeds of scanner data that account for almost 50 per cent of the food and drink market. 

The ONS are continuing the process to acquire data from further retailers to continually increase the coverage. This would be a significant step. As the ONS itself has said that the changes would “increase the number of price points dramatically each month from 180,000 to hundreds of millions”. Having access to such rich data will potentially allow the ONS to produce much more reliable high-frequency indices, particularly for food items. This would give the Bank of England for example higher quality indications of inflation trends when setting interest rates.

However, calculating such indices is not just a matter of having access to the right data. Traditional methods of calculating inflation rates can give quite odd answers when used to calculate high-frequency price changes with millions of products. Problems arise because individual products appear and disappear from one month to the next. Spending on goods can be volatile across time, and individual products may be heavily discounted before they disappear from the shelves – distorting the price index. 

Holding the basket of goods fixed across months in an environment with significant product churn can lead to an index that rapidly becomes unrepresentative of actual spending. Chaining the basket at for example monthly frequency can lead to implausible – indeed explosive – estimates of prices changes that are much greater than what one would obtain from direct comparison, a phenomenon known as “chain drift”.

One way to solve these problems is to adapt methods used to compare prices across countries: specifically, multilateral price indices. Multilateral price indices are more stable than standard bilateral indices used in the CPI because they compare the cost of purchasing a basket of goods across all periods within a window rather than just two. This mitigates the problem of seasonal and transitory products which might disappear from one month to the next. They also have the attractive feature that the estimated price change from January to March is the same as the product of price changes from January to February and from February to March: a basic property that is not shared by more traditional chained indices.

Subscribe to BusinessThink for the latest research, analysis and insights from UNSW Business School

There are a variety of multilateral indices which all share these basic features, but can also give different estimates of price changes. In addition, indices have to be updated as new price data is collected and the only way to do this without affecting measures of past price changes is to splice multilateral index numbers calculated over different windows (time periods) together. This reintroduces a limited amount of chain drift, albeit much smaller than that seen for traditional methods.

At present there is no real consensus as to which combination of approaches is the best way to calculate a multilateral index. Our ESCoE Discussion Paper assesses the theoretical properties and degree of chain drift exhibited by different multilateral index number methods, given different window lengths and using different splicing methods. We make the case for national statistical agencies to adopt the Caves-Christensen-Diewert-Inklaar index using the “mean splice” and a window length of at least 25 months.

Many questions remain, however – for example, how exactly should we deal with the prices of products that are only available for certain months of the year? And what should national statistical agencies do with other sources of data (such as web-scraped data) that don’t come with consumer expenditure information needed to weight items according to their importance? Outstanding questions like these need to be answered if national statistical agencies are to take maximum advantage of the data revolution.

Kevin Fox is Director of Centre for the Applied Economic Research (CAER) UNSW Business School, Peter Levell is an Associate Director at the Institute for Fiscal Studies (IFS) and Martin O’Connell is an Assistant Professor in the Economics Department at the University of Wisconsin-Madison. Republished with permission from New data and new challenges for Consumer Price Indices.