SGORNN: Combining scalar gates and orthogonal constraints in recurrent networks. (February 2023)

• Record Type: Journal Article
• Authors: Taylor-Melanson, Will; Ferreira, Martha Dais; Matwin, Stan
• Is Part Of: Neural networks. Volume 159(2023)
• Page Start: 25
• Page End: 33
• Publication Date: 2023-02
• Subjects: Recurrent Neural Networks -- Deep learning -- Unitary matrices -- Exploding gradient problem; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2022.11.028
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: Recurrent Neural Network (RNN) models have been applied in different domains, producing high accuracies on time-dependent data. However, RNNs have long suffered from exploding gradients during training, mainly due to their recurrent process. In this context, we propose a variant of the scalar gated FastRNN architecture, called Scalar Gated Orthogonal Recurrent Neural Networks (SGORNN). SGORNN utilizes orthogonal matrices at the recurrent step. Our experiments evaluate SGORNN using two recently proposed orthogonal parametrizations for the recurrent weights of an RNN. We present a constraint on the scalar gates of SGORNN, which is easily enforced at training time to provide a probabilistic generalization gap which grows linearly with the length of sequences processed. Next, we provide bounds on the gradients of SGORNN to show the impossibility of exponentially exploding gradients through time. Our experimental results on the addition problem confirm that our combination of orthogonal and scalar gated RNNs are able to outperform other orthogonal RNNs and LSTM on long sequences. We further evaluate SGORNN on the HAR-2 classification task, where it improves upon the accuracy of several models using far fewer parameters than standard RNNs. Finally, we evaluate SGORNN on the Penn Treebank word-level language modeling task, where it again outperforms its related architectures and shows comparable performance to LSTM using far less parameters. Overall, SGORNN shows higher representation capacity than the other orthogonal RNNs tested, suffers from less overfitting than other models in our experiments, benefits from a decrease in parameter count, and alleviates exploding gradients during backpropagation through time. Graphical abstract: Highlights: Both scalar-weighted residual connections and orthogonal transformations are used. A linear growth generalization gap bound is presented. Constraints required for generalization are easily enforced in practice. Exponentially exploding gradients through time are provably avoided. Gradient control and generalization are also observed in experiments.