SplitSR: An End-to-End Approach to Super-Resolution on Mobile Devices

Super-resolution (SR) is a coveted image processing technique for mobile apps ranging from the basic camera apps to mobile health. Existing SR algorithms rely on deep learning models with significant memory requirements, so they have yet to be deployed on mobile devices and instead operate in the cloud to achieve feasible inference time.This shortcoming prevents existing SR methods from being used in applications that require near real-time latency. In this work, we demonstrate state-of-the-art latency and accuracy for on-device super-resolution using a novel hybrid architecture called SplitSR and a novel lightweight residual block called SplitSRBlock. The SplitSRBlock supports channel-splitting, allowing the residual blocks to retain spatial information while reducing the computation in the channel dimension. SplitSR has a hybrid design consisting of standard convolutional blocks and lightweight residual blocks, allowing people to tune SplitSR for their computational budget. We evaluate our system on a low-end ARM CPU, demonstrating both higher accuracy and up to 5 times faster inference than previous approaches. We then deploy our model onto a smartphone in an app called ZoomSR to demonstrate the first-ever instance of on-device, deep learning-based SR. We conducted a user study with 15 participants to have them assess the perceived quality of images that were post-processed by SplitSR. Relative to bilinear interpolation -- the existing standard for on-device SR -- participants showed a statistically significant preference when looking at both images (Z=-9.270, p<0.01) and text (Z=-6.486, p<0.01).

SplitSR：一种在移动设备上实现超分辨率的端到端方法

超分辨率（SR）是一种令人垂涎的图像处理技术，适用于从基本相机应用到移动健康的各种移动应用。现有的SR算法依赖于具有大量内存需求的深度学习模型，因此它们尚未部署在移动设备上，而是在云中运行以实现可行的推理时间。.. 此缺点使现有的SR方法无法用于需要近实时延迟的应用程序中。在这项工作中，我们使用称为SplitSR的新型混合体系结构和称为SplitSRBlock的新型轻量残差块，演示了设备上超分辨率的最新延迟和准确性。SplitSRBlock支持通道拆分，从而允许残留块保留空间信息，同时减少了通道维的计算。SplitSR具有由标准卷积块和轻量残差块组成的混合设计，使人们可以根据自己的计算预算调整SplitSR。我们在低端ARM CPU上评估我们的系统，证明其精度更高，推理速度比以前的方法快5倍。然后，我们在名为ZoomSR的应用程序中将模型部署到智能手机上，以演示设备上基于深度学习的SR的第一个实例。我们与15名参与者进行了一项用户研究，以让他们评估由SplitSR后处理的图像的感知质量。相对于双线性插值-设备上SR的现有标准-参与者在查看图像（Z = -9.270，p <0.01）和文本（Z = -6.486，p <0.01）时均表现出统计学上的显着偏好。 （阅读更多）