Transformers for machine learning : a deep dive /: a deep dive. (2022)

• Record Type: Book
• Further Information: Note: Uday Kamath, Kenneth Graham, Wael Emara.
• Authors: Kamath, Uday; Graham, Kenneth L; Emara, Wael
• Contents: List of Figures ; List of Tables ; Author Bios ; Foreword ; Preface ; Contributors Deep Learning and Transformers: An Introduction ; 1.1 DEEP LEARNING: A HISTORIC PERSPECTIVE; 1.2 TRANSFORMERS AND TAXONOMY ; 1.2.1 Modified Transformer Architecture ; 1.2.1.1 Transformer block changes ; 1.2.1.2 Transformer sublayer changes ; 1.2.2 Pretraining Methods and Applications ; 1.3 RESOURCES ; 1.3.1 Libraries and Implementations ; 1.3.2 Books ; 1.3.3 Courses, Tutorials, and Lectures ; 1.3.4 Case Studies and Details Transformers: Basics and Introduction ; 2.1 ENCODER-DECODER ARCHITECTURE ; 2.2 SEQUENCE TO SEQUENCE ; 2.2.1 Encoder ; 2.2.2 Decoder ; 2.2.3 Training ; 2.2.4 Issues with RNN-based Encoder Decoder ; 2.3 ATTENTION MECHANISM ; 2.3.1 Background ; 2.3.2 Types of Score-Based Attention ; 2.3.2.1 Dot Product (multiplicative) ; 2.3.2.2 Scaled Dot Product or multiplicative ; 2.3.2.3 Linear, MLP, or additive ; 2.3.3 Attention-based Sequence to Sequence ; 2.4 TRANSFORMER ; 2.4.1 Source and Target Representation ; 2.4.1.1 Word Embedding ; 2.4.1.2 Positional Encoding ; 2.4.2 Attention Layers ; 2.4.2.1 Self-Attention ; 2.4.2.2 Multi-Head Attention ; 2.4.2.3 Masked Multi-Head Attention ; 2.4.2.4 Encoder-Decoder Multi-Head Attention ; 2.4.3 Residuals and Layer Normalization ; 2.4.4 Position-wise Feed-Forward Networks ; 2.4.5 Encoder ; 2.4.6 Decoder ; 2.5 CASE STUDY: MACHINE TRANSLATION ; 2.5.1 Goal ; 2.5.2 Data, Tools and Libraries ; 2.5.3 Experiments, Results and Analysis ; 2.5.3.1 Exploratory Data Analysis ; 2.5.3.2 Attention ; 2.5.3.3 Transformer; 2.5.3.4 Results and Analysis ; 2.5.3.5 Explainability Bidirectional Encoder Representations from Transformers (BERT) ; 3.1 BERT ; 3.1.1 Architecture ; 3.1.2 Pre-training ; 3.1.3 Fine-tuning ; 3.2 BERT VARIANTS ; 3.2.1 RoBERTa ; 3.3 APPLICATIONS ; 3.3.1 TaBERT ; 3.3.2 BERTopic ; 3.4 BERT INSIGHTS ; 3.4.1 BERT Sentence Representation ; 3.4.2 BERTology ; 3.5 CASE STUDY: TOPIC MODELING WITH TRANSFORMERS ; 3.5.1 Goal ; 3.5.2 Data, Tools, and Libraries ; 3.5.2.1 Data ; 3.5.2.2 Compute embeddings ; 3.5.3 Experiments, Results, and Analysis ; 3.5.3.1 Building Topics ; 3.5.3.2 Topic size distribution ; 3.5.3.3 Visualization of topics ; 3.5.3.4 Content of topics ; 3.6 CASE STUDY: FINE-TUNING BERT ; 3.6.1 Goal ; 3.6.2 Data, Tools and Libraries ; 3.6.3 Experiments, Results and Analysis Multilingual Transformer Architectures ; 4.1 MULTILINGUAL TRANSFORMER ARCHITECTURES ; 4.1.1 Basic Multilingual Transformer ; 4.1.2 Single-Encoder Multilingual NLU ; 4.1.2.1 mBERT ; 4.1.2.2 XLM ; 4.1.2.3 XLM-RoBERTa ; 4.1.2.4 ALM ; 4.1.2.5 Unicoder ; 4.1.2.6 INFOXL; 4.1.2.7 AMBER ; 4.1.2.8 ERNIE-M ; 4.1.2.9 HITCL ; 4.1.3 Dual-Encoder Multilingual NLU ; 4.1.3.1 LaBSE ; 4.1.3.2 mUSE ; 4.1.4 Multilingual NLG ; 4.2 MULTILINGUAL DATA ; 4.2.1 Pre-training Data ; 4.2.2 Multilingual Benchmarks ; 4.2.2.1 Classification ; 4.2.2.2 Structure Prediction ; 4.2.2.3 Question Answering ; 4.2.2.4 Semantic Retrieval ; 4.3 MULTILINGUAL TRANSFER LEARNING INSIGHTS ; 4.3.1 Zero-shot Cross-lingual Learning ; 4.3.1.1 Data Factors ; 4.3.1.2 Model Architecture Factors ; 4.3.1.3 Model Tasks Factors ; 4.3.2 Language-agnostic Cross-lingual Representations; 4.4 CASE STUDY ; 4.4.1 Goal ; 4.4.2 Data, Tools, and Libraries ; 4.4.3 Experiments, Results, and Analysis ; 4.4.3.1 Data Preprocessing ; 4.4.3.2 Experiments Transformer Modifications ; 5.1 TRANSFORMER BLOCK MODIFICATIONS ; 5.1.1 Lightweight Transformers ; 5.1.1.1 Funnel-Transformer ; 5.1.1.2 DeLighT ; 5.1.2 Connections between Transformer Blocks ; 5.1.2.1 RealFormer ; 5.1.3 Adaptive Computation Time ; 5.1.3.1 Universal Transformers (UT) ; 5.1.4 Recurrence Relations between Transformer Blocks ; 5.1.4.1 Transformer-XL ; 5.1.5 Hierarchical Transformers ; 5.2 TRANSFORMERS WITH MODIFIED MULTI-HEAD SELF-ATTENTION; 5.2.1 Structure of Multi-head Self-Attention ; 5.2.1.1 Multi-head self-attention ; 5.2.1.2 Space and time complexity ; 5.2.2 Reducing Complexity of Self-attention ; 5.2.2.1 Longformer ; 5.2.2.2 Reformer ; 5.2.2.3 Performer ; 5.2.2.4 Big Bird ; 5.2.3 Improving Multi-head-attention ; 5.2.3.1 Talking-Heads Attention ; 5.2.4 Biasing Attention with Priors ; 5.2.5 Prototype Queries; 5.2.5.1 Clustered Attention ; 5.2.6 Compressed Key-Value Memory ; 5.2.6.1 Luna: Linear Unified Nested Attention ; 5.2.7 Low-rank Approximations; 5.2.7.1 Linformer ; 5.3 MODIFICATIONS FOR TRAINING TASK EFFICIENCY ; 5.3.1 ELECTRA; 5.3.1.1 Replaced token detection ; 5.3.2 T5 ; 5.4 TRANSFORMER SUBMODULE CHANGES ; 5.4.1 Switch Transformer ; 5.5 CASE STUDY: SENTIMENT ANALYSIS; 5.5.1 Goal ; 5.5.2 Data, Tools, and Libraries ; 5.5.3 Experiments, Results, and Analysis ; 5.5.3.1 Visualizing attention head weights ; 5.5.3.2 Analysis Pretrained and Application-Specific Transformers ; 6.1 TEXT PROCESSING ; 6.1.1 Domain-Specific Transformers ; 6.1.1.1 BioBERT ; 6.1.1.2 SciBERT ; 6.1.1.3 FinBERT ; 6.1.2 Text-to-text Transformers ; 6.1.2.1 ByT5 ; 6.1.3 Text generation ; 6.1.3.1 GPT: Generative Pre-training ; 6.1.3.2 GPT-2 ; 6.1.3.3 GPT-3 ; 6.2 COMPUTER VISION ; 6.2.1 Vision Transformer ; 6.3 AUTOMATIC SPEECH RECOGNITION ; 6.3.1 Wav2vec 2.0 ; 6.3.2 Speech2Text2 ; 6.3.3 HuBERT: Hidden Units BERT ; 6.4 MULTIMODAL AND MULTITASKING TRANSFORMER ; 6.4.1 Vision-and-Language BERT (VilBERT) ; 6.4.2 Unified Transformer (UniT) ; 6.5 VIDEO PROCESSING WITH TIMESFORMER ; 6.5.1 Patch embeddings ; 6.5.2 Self-attention ; 6.5.2.1 Spatiotemporal self-attention ; 6.5.2.2 Spatiotemporal attention blocks ; 6.6 GRAPH TRANSFORMERS ; 6.6.1 Positional encodings in a graph ; 6.6.1.1 Laplacian positional encodings ; 6.6.2 Graph transformer input ; 6.6.2.1 Graphs without edge attributes ; 6.6.2.2 Graphs with edge attributes ; 6.7 REINFORCEMENT LEARNING ; 6.7.1 Decision Transformer ; 6.8 CASE STUDY: AUTOMATIC SPEECH RECOGNITION ; 6.8.1 Goal ; 6.8.2 Data, Tools, and Libraries ; 6.8.3 Experiments, Results, and Analysis ; 6.8.3.1 Preprocessing speech data ; 6.8.3.2 Evaluation Interpretability and Explainability Techniques for Transformers ; 7.1 TRAITS OF EXPLAINABLE SYSTEMS ; 7.2 RELATED AREAS THAT IMPACT EXPLAINABILITY ; 7.3 EXPLAINABLE METHODS TAXONOMY ; 7.3.1 Visualization Methods ; 7.3.1.1 Backpropagation-based ; 7.3.1.2 Perturbation-based ; 7.3.2 Model Distillation ; 7.3.2.1 Local Approximation ; 7.3.2.2 Model Translation ; 7.3.3 Intrinsic Methods ; 7.3.3.1 Probing Mechanism ; 7.3.3.2 Joint Training ; 7.4 ATTENTION AND EXPLANATION ; 7.4.1 Attention is not Explanation ; 7.4.1.1 Attention Weights and Feature Importance ; 7.4.1.2 Counterfactual Experiments ; 7.4.2 Attention is not not Explanation ; 7.4.2.1 Is attention necessary for all tasks? ; 7.4.2.2 Searching for Adversarial Models ; 7.4.2.3 Attention Probing ; 7.5 QUANTIFYING ATTENTION FLOW ; 7.5.1 Information flow as DAG ; 7.5.2 Attention Rollout ; 7.5.3 Attention Flow ; 7.6 CASE STUDY: TEXT CLASSIFICATION WITH EXPLAINABILITY ; 7.6.1 Goal ; 7.6.2 Data, Tools, and Libraries ; 7.6.3 Experiments, Results and Analysis ; 7.6.3.1 Exploratory Data Analysis ; 7.6.3.2 Experiments ; 7.6.3.3 Error Analysis and Explainability ; ; Bibliography ; Alphabetical Index
• Edition: 1st
• Publisher Details: Boca Raton : Chapman & Hall/CRC
• Publication Date: 2022
• Extent: 1 online resource, illustrations (black and white, and colour)
• Subjects: 006.32; Neural networks (Computer science); Computational intelligence; Machine learning
• Languages: English
• ISBNs: 9781000587098; 9781000587074; 9781003170082
• Related ISBNs: 9780367771652; 9780367767341
• Notes: Note: Includes bibliographical references and index.; Note: Description based on CIP data; resource not viewed.
• Access Rights: Legal Deposit; Only available on premises controlled by the deposit library and to one user at any one time; The Legal Deposit Libraries (Non-Print Works) Regulations (UK).
• Access Usage: Restricted: Printing from this resource is governed by The Legal Deposit Libraries (Non-Print Works) Regulations (UK) and UK copyright law currently in force.