Training a multilayer network with low-memory kernel-and-range projection. Issue 1 (January 2020)

• Record Type: Journal Article
• Authors: Zhuang, Huiping; Lin, Zhiping; Toh, Kar-Ann
• Is Part Of: Journal of the Franklin Institute. Volume 357:Issue 1(2020)
• Page Start: 522
• Page End: 550
• Publication Date: 2020-01
• Subjects: Science -- Periodicals; Technology -- Periodicals; Patents -- United States -- Periodicals; 505
• DOI: 10.1016/j.jfranklin.2019.11.074
• Languages: English
• ISSNs: 0016-0032

Abstract

Highlights: A low-memory formulation is proposed to address the high memory demand of a recently proposed kernel-and-range method. The proposed formulation is proved to be mathematically equivalent to the original kernel-and-range method. A regularization technique is introduced to cope with the existing rounding-error effect. Experiments including synthetic and real-world datasets are provided. Abstract: Recently, a learning method based on the kernel and the range space projections has been proposed. This method has been applied to learn the multilayer network analytically with interpretable relationships among the weight matrices. However, the bulk matrix based formulation suffers from a high memory demand during network learning. In this study, a low-memory resolution is proposed to address the memory demanding problem. Essentially, the bulk matrix operations are implemented by a low-memory formulation in which only one training sample is processed at a time. Such a formulation is proved to be mathematically equivalent to the original batch learning version. We also point out that the rounding errors in computing systems could hinder the performance of the proposed formulation. This formulation is then robustified by introducing a regularization technique with the cost of an additional but negligible memory usage. Our experiments show that the proposed low-memory resolution can indeed tremendously reduce the memory consumption while maintaining reasonably good performances in both regression and classification tasks.