TREE-STRUCTURED PARZEN ESTIMATORS WITH UNCERTAINTY FOR HYPERPARAMETER OPTIMIZATION OF MACHINE LEARNING ALGORITHMS

Abstract

Hyperparameter optimization (HPO) is one of the first tasks to be performed during the application of Machine Learning (ML) algorithms to real problems. Tree-structured Parzen estimators (TPE) have demonstrated their ability to find hyperparameter configurations in high dimensions with efficient evaluation budgets. However, as is common in HPO procedures, TPE ignores the fact that the expected performance of the algorithm, for any given HPO configuration, is affected by uncertainty. Building on the TPE algorithm proposed by Bergstra et al. (2011), we propose a strategy to account for this uncertainty and show that its management leads to better algorithm performance.