On the use of Random Forest to impute categorical variables beyond the sample

Abstract

The new paradigm of the Official Statistical production is based on a system of registers, resulting from the integration of administrative and survey data. Administrative data provide a complete enumeration of the units they cover: however, this population usually represents only a specific subset of the statistical target population, and such a subset is typically not obtained through a probabilistic sampling design. Similarly, survey data also cover only a subset of the population of interest, but they are made representative through the use of weights. Another common limitation of administrative sources is the delay in data availability.

In this context, generating a complete and consistent dataset is a critical task, which requires the implementation of specific procedures to account for delayed data and to impute missing values. One possible strategy is to use survey data as the target variable. However, this raises the issue of how to properly incorporate weights into the estimation model.

An example of a mass imputation approach is provided by the official estimates of the Attained Level of Education (ALE) adopted by the Italian National Institute of Statistics (Istat) for all the resident population in Italy. The official procedure is based on the estimation of different log-linear models.

In this application, we focus on the use of Random Forest (RF) to leverage the opportunities presented by Machine Learning (ML) technique on using all available information, including longitudinal data and variables with many categories. This approach allows for capturing complex relationships between variables, which is often challenging to incorporate comprehensively using standard methods.

The results are evaluated by focusing on different scenarios, each related to a level of available information, and focusing on three population subsets, each characterized by a different pattern of available information