The frequent insignificance of a "significant" p‐value. Issue 11 (3rd September 2021)

• Record Type: Journal Article
• Authors: McGiffin, David C.; Cumming, Geoff; Myles, Paul S.
• Is Part Of: Journal of cardiac surgery. Volume 36:Issue 11(2021)
• Page Start: 4322
• Page End: 4331
• Publication Date: 2021-09-03
• Subjects: Bayes theorem -- clinical review -- null hypothesis -- p values -- replication -- significance testing; Heart -- Surgery -- Periodicals; 617.412005
• DOI: 10.1111/jocs.15960
• Languages: English
• ISSNs: 0886-0440

Abstract

Abstract: Null hypothesis significance testing (NHST) and p ‐values are widespread in the cardiac surgical literature but are frequently misunderstood and misused. The purpose of the review is to discuss major disadvantages of p ‐values and suggest alternatives. We describe diagnostic tests, the prosecutor's fallacy in the courtroom, and NHST, which involve inter‐related conditional probabilities, to help clarify the meaning of p ‐values, and discuss the enormous sampling variability, or unreliability, of p ‐values. Finally, we use a cardiac surgical database and simulations to explore further issues involving p ‐values. In clinical studies, p ‐values provide a poor summary of the observed treatment effect, whereas the three‐number summary provided by effect estimates and confidence intervals is more informative and minimizes over‐interpretation of a "significant" result. p ‐values are an unreliable measure of the strength of evidence; if used at all they give only, at best, a very rough guide to decision making. Researchers should adopt Open Science practices to improve the trustworthiness of research and, where possible, use estimation (three‐number summaries) or other better techniques.