Harmonised data collection of ranking indicators: the Flemish ranking initiative

Abstract

Over the past decade, university rankings have become increasingly popular among a broad variety of end-users. Although ranking results, typically presented by a single rank score, seem to be easily interpretable, many caveats can be found in the indicators, their accompanying semantics as well as the methodologies used for data collection. In many cases, the semantic content of the indicators is poorly defined, allowing universities to collect data according to their own interpretation. This, together with differences in the local context of a university and the restrictions of data storage, results in a great variety in data that impedes a reliable comparison of universities. To overcome this variability, a Flemish initiative led by ECOOM-UHasselt, accommodated the semantic interpretation of the indicators used in several leading university rankings. In consultation with the ranking organisations, indicator definitions were scrutinised with regards to the specific interpretation of each of the four participating Flemish universities. Consequently, joint decisions with regards to the semantic interpretation and the data collection were taken and registered in a so-called ‘Flemish handbook for semantic comparable ranking results’. This level of fine-tuning ensures an identical interpretation and a harmonised data collection resulting in truly semantically comparable ranking results.