Essays on Machine Learning: Advances in Forecasting and Optimization

Abstract

The application of Machine Learning (ML) algorithms to many real-world problems poses several challenges such as the need for high-quality data, robustness, model interpretability, and high computational costs. These challenges require a multidisciplinary approach and ongoing research to enhance accuracy, transparency, and efficiency. Hyperparameters, set before training, significantly impact ML algorithm performance. Robust and efficient hyperparameter optimization (HPO) is essential due to the computational costs involved. Similar challenges arise in industrial applications with multi-objective, constrained, expensive to evaluate, and uncertain optimization objectives. Traditional evolutionary approaches are unsuitable for solving these problems, as they require a prohibitive number of experiments for evaluation. In the context of power forecasting for windmills, the short availability of data and the dynamic behavior of the system make traditional Recurrent Neural Networks (RNNs) less feasible to solve this forecasting problem. This thesis aims to develop efficient algorithms to address common challenges in Forecasting and Optimization, such as time constraints, data sparsity, and uncertainty. A multi-objective HPO algorithm is proposed, combining Multi-objective Tree Parzen Estimators (MOTPE) and Gaussian Process Regression (GPR) trained with heterogeneous noise. Additionally, the optimization algorithm based on Tree Parzen Estimators is modified to directly account for the performance variability in single objective HPO. Other contributions presented in this thesis are related to the application of GPR to emulate objectives and constraint functions in optimizing an adhesive bonding process with limited experimental data. The suggested Bayesian Optimization framework efficiently identifies optimal process settings with minimal physical experiments. Finally, a pipeline based on the Long Short-term Cognitive Network (LSTCN) was designed to address the challenges of data volatility and high processing time of windmill power forecasting. The results showed in this thesis exhibited low forecasting errors and training/testing times compared to other recurrent models. Hence, the superiority of the proposed forecasting pipeline.