Advertisement

Generating Rare Surgical Events Using CycleGAN: Addressing Lack of Data for Artificial Intelligence Event Recognition

Published:November 25, 2022DOI:https://doi.org/10.1016/j.jss.2022.11.008

      Abstract

      Introduction

      Artificial Intelligence (AI) has shown promise in facilitating surgical video review through automatic recognition of surgical activities/events. There are few public video data sources that demonstrate critical yet rare events which are insufficient to train AI for reliable video event recognition. We suggest that a generative AI algorithm can create artificial massive bleeding images for minimally invasive lobectomy that can be used to augment the current lack of data in this field.

      Materials and methods

      A generative adversarial network (GAN) algorithm was used (CycleGAN) to generate artificial massive bleeding event images. To train CycleGAN, six videos of minimally invasive lobectomies were utilized from which 1819 frames of nonbleeding instances and 3178 frames of massive bleeding instances were used.

      Results

      The performance of the CycleGAN algorithm was tested on a new video that was not used during the training process. The trained CycleGAN was able to alter the laparoscopic lobectomy images according to their corresponding massive bleeding images, where the contents of the original images were preserved (e.g., location of tools in the scene) and the style of each image is changed to massive bleeding (i.e., blood automatically added to appropriate locations on the images).

      Conclusions

      The result could suggest a promising approach to supplement the lack of data for the rare massive bleeding event that can occur during minimally invasive lobectomy. Future work could be dedicated to developing AI algorithms to identify surgical strategies and actions that potentially lead to massive bleeding and warn surgeons prior to this event occurrence.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Surgical Research
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Korndorffer J.R.
        • Hawn M.T.
        • Spain D.A.
        • et al.
        Situating artificial intelligence in surgery: a focus on disease severity.
        Ann Surg. 2020; 272: 523-528
        • Maier-Hein L.
        • Vedula S.S.
        • Speidel S.
        • et al.
        Surgical data science for next-generation interventions.
        Nat Biomed Eng. 2017; 1: 691-696
        • Grogan E.L.
        • Jones D.R.
        VATS lobectomy is better than open thoracotomy: what is the evidence for short-term outcomes?.
        Thorac Surg Clin. 2008; 18: 249-258
        • Lim E.
        • Batchelor T.
        • Shackcloth M.
        • et al.
        Study protocol for VIdeo assisted thoracoscopic lobectomy versus conventional Open LobEcTomy for lung cancer, a UK multicentre randomised controlled trial with an internal pilot (the VIOLET study).
        BMJ Open. 2019; 9: e029507
        • Desai H.
        • Natt B.
        • Kim S.
        • Bime C.
        Decreased in-hospital mortality after lobectomy using video-assisted thoracoscopic surgery compared with open thoracotomy.
        Ann Am Thorac Soc. 2017; 14: 262-266
        • Gonzalez-Rivas D.
        • Stupnik T.
        • Fernandez R.
        • et al.
        Intraoperative bleeding control by uniportal video-assisted thoracoscopic surgery †.
        Eur J Cardiothorac Surg. 2016; 49: i17-i24
        • Kent M.
        • Wang T.
        • Whyte R.
        • Curran T.
        • Flores R.
        • Gangadharan S.
        Open, video-assisted thoracic surgery, and robotic lobectomy: review of a national database.
        Ann Thorac Surg. 2014; 97 (discussion 242-244): 236-242
        • Goodfellow I.J.
        • Pouget-Abadie J.
        • Mirza M.
        • et al.
        Generative adversarial networks. ArXiv14062661 Cs stat.
        (Available at:)
        http://arxiv.org/abs/1406.2661
        Date: 2014
        Date accessed: January 7, 2021
        • Zhu J.Y.
        • Park T.
        • Isola P.
        • Efros A.A.
        Unpaired image-to-image translation using cycle-consistent adversarial networks. ArXiv170310593 Cs.
        (Available at:)
        http://arxiv.org/abs/1703.10593
        Date: 2020
        Date accessed: January 7, 2021
        • Schmidt V.
        • Luccioni A.
        • Mukkavilli S.K.
        • et al.
        Visualizing the consequences of climate change using cycle-consistent adversarial networks. ArXiv190503709 Cs.
        (Available at:)
        http://arxiv.org/abs/1905.03709
        Date: 2019
        Date accessed: January 7, 2021
        • Novellis P.
        • Jadoon M.
        • Cariboni U.
        • Bottoni E.
        • Pardolesi A.
        • Veronesi G.
        Management of robotic bleeding complications.
        Ann Cardiothorac Surg. 2019; 8: 292-295
        • Dunning J.
        • Walker W.S.
        Pulmonary artery bleeding caused during VATS lobectomy.
        Ann Cardiothorac Surg. 2012; 1: 109-110
        • Hashimoto D.A.
        • Rosman G.
        • Witkowski E.R.
        • et al.
        Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy.
        Ann Surg. 2019; 270: 414-421
        • Jin Y.
        • Dou Q.
        • Chen H.
        • et al.
        SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network.
        IEEE Trans Med Imaging. 2018; 37: 1114-1126
        • Jin Y.
        • Li H.
        • Dou Q.
        • et al.
        Multi-task recurrent convolutional network with correlation loss for surgical video analysis.
        Med Image Anal. 2020; 59: 101572
        • Twinanda A.P.
        • Shehata S.
        • Mutter D.
        • Marescaux J.
        • de Mathelin M.
        • Padoy N.
        EndoNet: a deep architecture for recognition tasks on laparoscopic videos.
        IEEE Trans Med Imaging. 2017; 36: 86-97
        • Mohamadipanah H.
        • Kearse L.
        • Witt A.
        • et al.
        Can deep learning algorithms help identify surgical workflow and techniques?.
        J Surg Res. 2021; 268: 318-325
        • Mohamadipanah H.
        • Perumalla C.
        • Yang S.
        • et al.
        Artificial intelligence in surgery: a research team perspective.
        Curr Probl Surg. 2022; 59: 101125
        • Lee K.
        • Choi M.K.
        • Jung H.
        Davincigan: unpaired surgical instrument translation for data augmentation. In: International Conference on Medical Imaging with Deep Learning.
        (Available at:)
        http://proceedings.mlr.press/v102/lee19a.html
        Date: 2019
        Date accessed: April 19, 2021
        • Engelhardt S.
        • De Simone R.
        • Full P.M.
        • Karck M.
        • Wolf I.
        Improving surgical training phantoms by hyperrealism: deep unpaired image-to-image translation from real surgeries.
        in: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham2018: 747-755
        • Xu J.
        • Anwar S.
        • Barnes N.
        • et al.
        Ofgan: realistic rendition of synthetic colonoscopy videos.
        in: Martel A.L. Abolmaesumi P. Stoyanov D. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Lecture Notes in Computer Science. Springer, Cham2020: 732-741
        • Ma R.
        • Reddy S.
        • Vanstrum E.B.
        • Hung A.J.
        Innovations in urologic surgical training.
        Curr Urol Rep. 2021; 22: 26
        • Judkins T.N.
        • Oleynikov D.
        • Stergiou N.
        Enhanced robotic surgical training using augmented visual feedback.
        Surg Innov. 2008; 15: 59-68
        • Rothermel L.D.
        • Lipman J.M.
        Estimation of blood loss is inaccurate and unreliable.
        Surgery. 2016; 160: 946-953