Cost-Efficient Online Hyperparameter Optimization
Recent work on hyperparameters optimization (HPO) has shown the possibility of training certain hyperparameters together with regular parameters. However, these online HPO algorithms still require running evaluation on a set of validation examples at each training step, steeply increasing the training cost.To decide when to query the validation loss, we model online HPO as a time-varying Bayesian optimization problem, on top of which we propose a novel \textit{costly feedback} setting to capture the concept of the query cost. Under this setting, standard algorithms are cost-inefficient as they evaluate on the validation set at every round. In contrast, the cost-efficient GP-UCB algorithm proposed in this paper queries the unknown function only when the model is less confident about current decisions. We evaluate our proposed algorithm by tuning hyperparameters online for VGG and ResNet on CIFAR-10 and ImageNet100. Our proposed online HPO algorithm reaches human expert-level performance within a single run of the experiment, while incurring only modest computational overhead compared to regular training.
具有成本效益的在线超参数优化
超参数优化（HPO）的最新工作表明，可以训练某些超参数以及常规参数。但是，这些在线HPO算法仍然需要在每个训练步骤中对一组验证示例进行评估，从而大大增加了训练成本。.. 为了决定何时查询验证损失，我们将在线HPO建模为时变贝叶斯优化问题，在此基础上，我们提出了一种新颖的\ textit {costly feedback}设置来捕获查询成本的概念。在这种设置下，标准算法在每次评估时都会对验证集进行评估，因此成本效率低下。相比之下，本文提出的具有成本效益的GP-UCB算法仅在模型对当前决策缺乏信心时才查询未知函数。我们通过在线调整CIFAR-10和ImageNet100上的VGG和ResNet的超参数来评估我们提出的算法。我们提出的在线HPO算法可在一次实验中达到人类专家级的性能，而与常规训练相比，仅产生适度的计算开销。 （阅读更多）
暂无评论