Validating a membership disclosure metric for synthetic health data. Issue 4 (11th October 2022)

• Record Type: Journal Article
• Authors: El Emam, Khaled; Mosquera, Lucy; Fang, Xi
• Is Part Of: JAMIA open. Volume 5:Issue 4(2022)
• Publication Date: 2022-10-11
• Subjects: synthetic data generation -- data privacy -- membership disclosure; Medical informatics -- Periodicals; 610.285
• DOI: 10.1093/jamiaopen/ooac083
• Languages: English
• ISSNs: 2574-2531

Abstract

Abstract: Background: One of the increasingly accepted methods to evaluate the privacy of synthetic data is by measuring the risk of membership disclosure. This is a measure of the F1 accuracy that an adversary would correctly ascertain that a target individual from the same population as the real data is in the dataset used to train the generative model, and is commonly estimated using a data partitioning methodology with a 0.5 partitioning parameter. Objective: Validate the membership disclosure F1 score, evaluate and improve the parametrization of the partitioning method, and provide a benchmark for its interpretation. Materials and methods: We performed a simulated membership disclosure attack on 4 population datasets: an Ontario COVID-19 dataset, a state hospital discharge dataset, a national health survey, and an international COVID-19 behavioral survey. Two generative methods were evaluated: sequential synthesis and a generative adversarial network. A theoretical analysis and a simulation were used to determine the correct partitioning parameter that would give the same F1 score as a ground truth simulated membership disclosure attack. Results: The default 0.5 parameter can give quite inaccurate membership disclosure values. The proportion of records from the training dataset in the attack dataset must be equal to the sampling fraction of the real dataset from the population. The approach is demonstrated on 7 clinical trial datasets. Conclusions: Our proposed parameterization, as well as interpretation and generative model training guidance provide a theoretically and empirically grounded basis for evaluating and managing membership disclosure risk for synthetic data. Lay Summary: Membership disclosure is considered an important type of privacy risk for synthetic data. A commonly applied methodology for evaluating membership disclosure is the partitioning method. We demonstrate theoretically and empirically through a simulation on 4 population datasets that current parameterizations of this method can potentially give inaccurate estimates of risk, and propose a more grounded parametrization. We further provide an interpretable version of that metric, a benchmark for deciding when membership disclosure is acceptably small, and a proposed metric to manage utility and membership disclosure risk during the training of generative models which generate the synthetic datasets. Finally, we demonstrate its application on 7 oncology clinical trial datasets.