Underestimation modification for intrinsic dimension estimation. (August 2023)

• Record Type: Journal Article
• Authors: Qiu, Haiquan; Yang, Youlong; Pan, Hua
• Is Part Of: Pattern recognition. Volume 140(2023)
• Publication Date: 2023-08
• Subjects: Intrinsic dimension -- Parameter selection -- Estimation method -- Underestimation modification -- Smooth manifold; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2023.109580
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: The upper and lower bounds of parameter κ in ID( κ ) algorithm are analyzed. Use search methods to determine intrinsic dimensionality and increase speed. underestimation is modified using simulation benchmarks to improve accuracy. Simulation and real datasets verify the accuracy and robustness of the estimate. Abstract: The intrinsic dimension is the dimension of the low-dimensional manifold where the high-dimensional data is located. Accurately estimating the intrinsic dimension of the data set is helpful for data-dimensionality reduction and preprocessing. Due to the unknown spatial distribution of data and the limited sample size of a dataset, estimation methods which only use distance information tend to underestimate the intrinsic dimension of dataset. To reduce the estimation complexity and improve the accuracy, two estimation algorithms based on ID( κ ) are proposed, where κ is the scaling ratio of the neighborhood radius of the sample point. First, according to the selection criteria of parameter κ, an improved algorithm for selecting the optimal scaling ratio κ is proposed, which reduces the computational complexity and improves the stability of estimation. Second, using simulation datasets with the same sample size and known intrinsic dimensions, the relationship between the estimated dimension and the true intrinsic dimension is obtained, and an underestimation modification method for intrinsic dimension estimation is proposed. Results of comparative experiments on simulation and real datasets indicate that the underestimation modification algorithm has high estimation accuracy and robustness.