MALI: A memory efficient and reverse accurate integrator for Neural ODEs

MALI: A memory efficient and reverse accurate integrator for Neural ODEs

Neural ordinary differential equations (Neural ODEs) are a new family of deep-learning models with continuous depth. However, the numerical estimation of the gradient in the continuous case is not well solved: existing implementations of the adjoint method suffer from inaccuracy in reverse-time trajectory, while the naive method and the adaptive checkpoint adjoint method (ACA) have a memory cost that grows with integration time.In this project, based on the asynchronous leapfrog (ALF) solver, we propose the Memory-efficient ALF Integrator (MALI), which has a constant memory cost $w.r.t$ integration time similar to the adjoint method, and guarantees accuracy in reverse-time trajectory (hence accuracy in gradient estimation). We validate MALI in various tasks: on image recognition tasks, to our knowledge, MALI is the first to enable feasible training of a Neural ODE on ImageNet and outperform a well-tuned ResNet, while existing methods fail due to either heavy memory burden or inaccuracy; for time series modeling, MALI significantly outperforms the adjoint method; and for continuous generative models, MALI achieves new state-of-the-art performance.

MALI：用于神经ODE的高效存储和反向精确积分器

神经常微分方程（Neural ODEs）是具有连续深度的新的深度学习模型系列。但是，连续情况下梯度的数值估计并不能很好地解决：伴随方法的现有实现存在逆时轨迹的不准确性，而朴素方法和自适应检查点伴随方法（ACA）的存储成本为随着集成时间的增长。.. 在此项目中，我们基于异步跨越式（ALF）求解器，提出了一种内存效率高的ALF集成器（MALI），它具有恒定的内存成本 w。[R。Ť 积分时间类似于伴随方法，并保证了逆时轨迹的准确性（因此，梯度估计的准确性）。我们在各种任务中验证MALI：就图像识别任务而言，据我们所知，MALI是第一个能够在ImageNet上对神经ODE进行可行的训练，并且性能优于经过良好调整的ResNet的方法，而现有方法则可能由于沉重的内存负担或不准确而失败; 对于时间序列建模，MALI的性能明显优于伴随方法。对于连续的生成模型，MALI实现了最新的性能。 （阅读更多）