LOW RANK APPROXIMATIONS FOR NEURAL NETWORKS
UDC 004.89
DOI:10.26102/2310-6018/2020.30.3.018
N.V. Shaposhnikova
Today, artificial neural networks (hereinafter ANN) and deep learning have become almost indispensable in applications related to the tasks of machine vision, machine translation, speech to text conversion, text rubrication, video processing, etc. However, despite the presence of a number of classical theorems substantiating the approximating capabilities of neural network structures, the current successes in the field of ANNs in most cases are associated with the heuristic construction of the network architecture applicable only for the specific problem under consideration. On the other hand, deep ANNs have millions of parameters and require powerful computing devices for their functioning, which limits the possibilities of their application, for example, on mobile devices. Significant progress in solving these problems can be obtained using modern powerful algorithms of low-rank approximations for the parameters of the ANN layers, which will both simplify the process of developing a neural network architecture and will lead to significant compression and acceleration of the training of deep ANNs. Considering, for example, the core of the convolutional ANN as a four-dimensional array (tensor), we can construct a low-rank approximation for it with the effective implementation of its convolution with the vector (direct signal propagation in the network when generating the prediction) and differentiation with respect to the parameters (back signal propagation in the network when training). In this paper, we will consider the modern paradigm of machine learning and low-rank tensor approximations, and we will demonstrate the prospects for the tensorization of deep ANNs using a specific model numerical example corresponding to the task of automatic recognition of handwritten digits.
Keywords: machine learning, neural network, deep convolutional network, low rank approximation.
Full text:
Shaposhnikova_3_20_1.pdf