On Optimal Correlation-Based Prediction. Issue 4 (2nd October 2022) - Bottai, Matteo; Kim, Taeho; Lieberman, Benjamin; Luta, George; Peña, Edsel
Abstract: This note examines, at the population-level, the approach of obtaining predictors h ˜ ( X ) of a random variable Y, given the joint distribution of ( Y, X ), by maximizing the mapping h ↦ κ ( Y, h ( X ) ) for a given correlation function κ ( ·, · ) . Commencing with Pearson's correlation function, the class of such predictors is uncountably infinite. The least-squares predictor h * is an element of this class obtained by equating the expectations of Y and h ( X ) to be equal and the variances of h ( X ) and E ( Y | X ) to be also equal. On the other hand, replacing the second condition by the equality of the variances of Y and h ( X ), a natural requirement for some calibration problems, the unique predictor h * * that is obtained has the maximum value of Lin's (1989 ) concordance correlation coefficient (CCC) with Y among all predictors. Since the CCC measures the degree of agreement, the new predictor h * * is called the maximal agreement predictor. These predictors are illustrated for three special distributions: the multivariate normal distribution; the exponential distribution, conditional on covariates; and the Dirichlet distribution. The exponential distribution is relevant in survival analysis or in reliability settings, while the Dirichlet distribution is relevant for compositional data.