O-120 Embryo ranking agreement between embryologists and AI algorithms. (30th June 2022)

• Record Type: Journal Article
• Authors: Zaninovic, N; Sierra, J; Malmsten, J; Rosenwaks, Z
• Is Part Of: Human reproduction. Volume 37(2022)Supplement 1
• Publication Date: 2022-06-30
• Subjects: Human reproduction -- Periodicals; 618
• DOI: 10.1093/humrep/deac105.020
• Languages: English
• ISSNs: 0268-1161

Abstract

Abstract: Study question: What is the level of agreement between different AI algorithms and embryologists when ranking blastocysts? Summary answer: In general, embryologists have a stronger level of agreement with each other, whereas AI algorithms differ greatly between embryologists and among each other. What is known already: Previous studies comparing agreement among embryologists ranking embryos have shown moderate to high inter-and intra-agreement levels. To our knowledge, this is the first study that endeavors to evaluate the level of agreement between different AI algorithms and embryologists in regard to ranking embryo quality. Study design, size, duration: Study data consisted of time-lapse images of 800 embryos from 100 patients (8 embryos each). All embryos were created from fresh oocytes retrieved at a single center between 2019 and 2020 and fertilized using ICSI. They were cultured in TLM incubators (Vitrolife, Sweden) and developed for 120 hours. The cohort included at least 8 embryos that started to blastulate (sTB) before 120 hours post-fertilization (HPF–ICSI). Patients older than 38 years were excluded. Participants/materials, setting, methods: Five international embryologists ranked embryos using single images; three also ranked embryos using TLM videos. Eight international AI companies anonymously ranked the embryos using AI models; half used single images while the others used full videos. The Kendal Tau statistic was used to determine the agreement level between the ranking methods; -1 denotes 100% disagreement and 1 denotes perfect agreement. The pair-wise agreement in selecting the top-one and top-two embryos was compared across all methods. Main results and the role of chance: The embryologists had relatively high degree of agreement in the overall ranking of 100 cycles (average K-t=0.70), slightly lower than the inter-embryologist agreement when using a single image or video (average K-t=0.78). Overall agreement between embryologists and the AI algorithms was significantly lower (average K-t=0.53) and similar to inter-AI algorithm agreement (average K-t=0.47). Notably, two of the eight algorithms had a very low agreement with other ranking methodologies (average K-t=0.05). The average agreement in selecting the best-quality embryo (1/8 in 100 cycles, expected agreement by random chance, 12.5% CI95 :6-19%) was 59.5% among embryologists and 40.3% for six AI algorithms, for the two algorithms with the low overall agreement, the incidence of the agreement was 11.7%. Agreement on selecting the same top-two embryos/cycle (expected agreement by random chance, 25.0% CI95 :17-32%) was 73.5% among embryologists and 56.0% among AI methods excluding two discordant algorithms, which had an average agreement of 24.4%, the expected range of agreement from random chance. Intra-embryologist ranking agreement (single image vs. video) was 71.7% and 77.8% for single and top-two embryos, respectively. Analysis of average raw scores indicated cycles with low diversity of embryo quality generally resulted in lower overall agreement between the methods (embryologists and AI models). Limitations, reasons for caution: Given the selection process for cycles and the corresponding embryos, the ground truth cannot be assertained as no implantation or pregnancy outcome was assessed or compared. Although this study can identify agreement between different ranking methods, it can not determine which assessment method is correct. Wider implications of the findings: Our results suggest that the AI method used to assign relative embryo quality may result in a significantly different selection and, presumably, outcome. Further studies should evaluate the source of the disagreement in embryos for which the outcome is known. Trial registration number: N/A