The probability of improved prediction as a new concept for model selection in the presence of outliers

Abstract

Robust regression techniques in combination with dedicated information criteria have proven to be successful in resolving model selection issues related to outlying data points. Often, the employed loss function in these criteria is made more robust through the consideration of a bounded version. In this paper, the probability of improved prediction (PIP) is proposed as an alternative that does not require the modification of the loss function. In general, the PIP is a probabilistic measure for directly comparing two competing models. Based on a user-defined loss function, this is achieved by quantifying the frequency of instances that one model gives better predictions than the other model. The simulation study shows comparable performance between the PIP and its competitors for selecting linear regression models. It is also shown how the PIP can be applied within more complicated machine learning models as well.