Web data sources and official statistical standards: the WIN experience

Abstract

In order to promote the integration of web data sources for official statistics, Eurostat has launched a four-year initiative, the Web Intelligence Network project. One of the main objectives of the project is to create a network within the European Statistical System (ESS) and to develop a common infrastructure, the Web Intelligence Hub (WIH), providing services and tools for web data collection and management. The WIH is designed to support National Statistical Institutes (NSIs) in all stages of web data collection and processing. In the long term, the WIH can be further developed to explore the potential of additional innovative data sources for official statistics. During the WIN project, the architectural task has dealt with the Enhancement and Enrichment (E&E) of the architectural standard Big Data REference Architecture and Layers (BREAL). BREAL is an architectural framework developed in the Big Data II project to support NSIs in planning their investments in big data. It provides a set of tools for defining the business objectives, application components and data models needed to develop statistical processes based on big data. One of the main outcomes achieved by the architectural task is the specialization of BREAL for web data. The adoption and E&E of the BREAL framework has enabled the harmonization between the requirements of the project use cases and the services provided by the WIH. BREAL specialization results from the combination of the project experience and it is intended to promote the deployment of web data workflows in the production environment. This enhancement of BREAL enables a NSI to assess the maturity level of a use case. In addition, the adoption of this approach increases process standardization and the development of shareable tools that can be reused by other statistical organizations, and/or adapted to other use cases and/or statistical domains.