Thomas Welchowski and Matthias Schmid - kernDeepStackNet: An R package for tuning kernel deep stacking networks
Department of Medical Biometry, Informatics and Epidemiology, University Bonn
Download the slides here
Kernel deep stacking networks (KDSNs) are a novel method for supervised learning in biomedical research. Belonging to the class of deep learning methods, KDSNs use multiple layers of non-linear transformations to derive abstractions of the input variables. This architecture can efficiently represent complex nonlinear dependencies in the joint distribution of the inputs and the response variable. While training of deep artificial neural networks usually involves the optimization of a non-convex problem, which often implies local optima and slow convergence, KDSNs are characterized by an efficient fitting procedure that is based on a series of kernel ridge regression models with closed-form solutions.
The talk will address the tuning of KDSNs, which is a challenging task due to the multiple hyper-parameters that have to be specified before network fitting. Specifically, we propose a data-driven tuning strategy for KDSNs that is based on model-based optimization (MBO). The proposed tuning approach explores the hyper-parameter space via a meta-model that uses a performance criterion such as the area under the curve (AUC) or the root mean squared error (RMSE) as outcome variable. Simulation studies show that the MBO approach is substantially faster than traditional grid search strategies. Analyses of real data sets demonstrate that MBO-tuned KDSNs are competetive to other state-of-art machine learning techniques in terms of prediction accuracy. We also extend the KDSN framework by new tools for variable selection and dropout. The fitting and tuning procedures are implemented in the R package kernDeepStackNet.