Bias-variance tradeoff in machine learning: Theoretical formulation and implications to structural engineering applications. (December 2022)

• Record Type: Journal Article
• Authors: Guan, Xingquan; Burton, Henry
• Is Part Of: Structures. Volume 46(2022)
• Page Start: 17
• Page End: 30
• Publication Date: 2022-12
• Subjects: Bias-variance decomposition -- Machine learning -- Hyperparameter selection -- Data-driven structural engineering -- Model complexity and generalizability; Structural engineering -- Periodicals; 624.1
• DOI: 10.1016/j.istruc.2022.10.004
• Languages: English
• ISSNs: 2352-0124

Abstract

Abstract: When machine learning (ML) is used to solve practical problems in structural engineering (SE), the associated data may not reflect the overall distribution of the population. With this in mind, a common practice in prior ML-SE studies is to optimize the model such that the testing error is minimized. However, this strategy can lead to a situation where the testing error is significantly greater than the error obtained during training, which is a clear sign of overfitting. Therefore, simply selecting a model with the highest testing performance does not necessarily guarantee generalizability. To address this challenge, this study aims to provide some practical guidelines that consider both the model complexity and generalizability. The proposed solution is fairly straightforward to implement. The generalized bias-variance decomposition is first introduced, followed by a discussion of different error variation patterns (bell-shaped, double-descent, and monotonically decreasing). A mathematical formulation that establishes a relationship between the model complexity and generalizability via an objective function is then presented to facilitate the model parameter selection. Subsequently, four numerical experiments are conducted to apply the formulation within the structural engineering domain. Finally, the proposed model selection approach is compared with existing methods to demonstrate its advantages and limitations.