AI-Based Video Segmentation: Procedural Steps or Basic Maneuvers?

Published:November 24, 2022DOI:



      Video-based review of surgical procedures has proven to be useful in training by enabling efficiency in the qualitative assessment of surgical skill and intraoperative decision-making. Current video segmentation protocols focus largely on procedural steps. Although some operations are more complex than others, many of the steps in any given procedure involve an intricate choreography of basic maneuvers such as suturing, knot tying, and cutting. The use of these maneuvers at certain procedural steps can convey information that aids in the assessment of the complexity of the procedure, surgical preference, and skill. Our study aims to develop and evaluate an algorithm to identify these maneuvers.


      A standard deep learning architecture was used to differentiate between suture throws, knot ties, and suture cutting on a data set comprised of videos from practicing clinicians (N = 52) who participated in a simulated enterotomy repair. Perception of the added value to traditional artificial intelligence segmentation was explored by qualitatively examining the utility of identifying maneuvers in a subset of steps for an open colon resection.


      An accuracy of 84% was reached in differentiating maneuvers. The precision in detecting the basic maneuvers was 87.9%, 60%, and 90.9% for suture throws, knot ties, and suture cutting, respectively. The qualitative concept mapping confirmed realistic scenarios that could benefit from basic maneuver identification.


      Basic maneuvers can indicate error management activity or safety measures and allow for the assessment of skill. Our deep learning algorithm identified basic maneuvers with reasonable accuracy. Such models can aid in artificial intelligence-assisted video review by providing additional information that can complement traditional video segmentation protocols.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Surgical Research
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Birkmeyer J.D.
        • Finks J.F.
        • O’Reilly A.
        • et al.
        Surgical skill and complication rates after bariatric surgery.
        N Engl J Med. 2013; 369: 1434-1442
        • Ibrahim A.M.
        • Varban O.A.
        • Dimick J.B.
        Novel uses of video to accelerate the surgical learning curve.
        J Laparoendosc Adv Surg Tech. 2016; 26: 240-242
      1. ABS to explore video-based assessment in pilot program launching june 2021 | American board of surgery.
        (Available at:)
        • Dath D.
        • Regehr G.
        • Birch D.
        • et al.
        Toward reliable operative assessment: the reliability and feasibility of videotaped assessment of laparoscopic technical skills.
        Surg Endosc. 2004; 18: 1800-1804
        • Garrow C.R.
        • Kowalewski K.F.
        • Li L.
        • et al.
        Machine learning for surgical phase recognition: a systematic review.
        Ann Surg. 2020; 273: 684-693
        • Jin Y.
        • Dou Q.
        • Chen H.
        • et al.
        SV-RCNet: workflow recognition from surgical videos using recurrent convolutional network.
        IEEE Trans Med Imaging. 2018; 37: 1114-1126
        • Korndorffer J.R.
        • Hawn M.T.
        • Spain D.A.
        • et al.
        Situating artificial intelligence in surgery: a focus on disease severity.
        Ann Surg. 2020; 272: 523-528
        • Hashimoto D.A.
        • Axelsson G.C.
        • Jones C.B.
        • et al.
        Surgical procedural map scoring for decision-making in laparoscopic cholecystectomy.
        Am J Surg. 2018; 217: 356-361
        • Mohamadipanah H.
        • Nathwani J.
        • Peterson K.
        • et al.
        Shortcut assessment: can residents’ operative performance be determined in the first five minutes of an operative task?.
        Surgery. 2018; 163: 1207-1212
        • Basiev K.
        • Goldbraikh A.
        • Pugh C.M.
        • Laufer S.
        Open surgery tool classification and hand utilization using a multi-camera system.
        Int J Comput Assist Radiol Surg. 2021; 17: 1497-1505
        • He K.
        • Zhang X.
        • Ren S.
        • Sun J.
        Deep residual learning for image recognition.
        (Available at:)
        Date: 2015
        Date accessed: April 16, 2021
        • Hochreiter S.
        • Schmidhuber J.
        Long short-term memory.
        Neural Comput. 1997; 9: 1735-1780
        • Deng J.
        • Dong W.
        • Socher R.
        • Li L.
        • Li K.
        • Fei-Fei L.
        ImageNet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition.
        IEEE. 2009; (Available at:): 248-255
        • Miskovic D.
        • Ni M.
        • Wyles S.
        • et al.
        Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England.
        Ann Surg. 2013; 257: 476-482
        • Schumpelick V.
        • Kasperk R.
        • Stumpf M.
        Atlas of General Surgery.
        Thieme, New York, NY2009
        • Ellison E.C.
        • Zollinger R.M.
        Colectomy, right.
        in: Zollinger’s Atlas of Surgical Operations, 10e. McGraw-Hill Education, 2016
        • Gehanno J.F.
        • Rollin L.
        • Le Jean T.
        • Louvel A.
        • Darmoni S.
        • Shaw W.
        Precision and recall of search strategies for identifying studies on return-to-work in medline.
        J Occup Rehabil. 2009; 19: 223-230
        • Vedula S.S.
        • Malpani A.O.
        • Tao L.
        • et al.
        Analysis of the structure of surgical activity for a suturing and knot-tying task.
        PLoS One. 2016; 11e0149174
        • Ahmidi N.
        • Gao Y.
        • Béjar B.
        • et al.
        String motif-based description of tool motion for detecting skill and gestures in robotic surgery.
        in: Salinesi C. Norrie M.C. Pastor Ó. Advanced Information Systems Engineering. Vol 7908. Lecture Notes in Computer Science. Springer Berlin Heidelberg, 2013: 26-33
        • Ahmidi N.
        • Poddar P.
        • Jones J.D.
        • et al.
        Automated objective surgical skill assessment in the operating room from unstructured tool motion in septoplasty.
        Int J Comput Assist Radiol Surg. 2015; 10: 981-991
        • MacKenzie L.
        • Ibbotson J.A.
        • Cao C.G.L.
        • Lomax A.J.
        Hierarchical decomposition of laparoscopic surgery: a human factors approach to investigating the operating room environment.
        Minim Invasive Ther Allied Technol. 2001; 10: 121-127
        • Lin H.C.
        • Shafran I.
        • Murphy T.E.
        • Okamura A.M.
        • Yuh D.D.
        • Hager G.D.
        Automatic detection and segmentation of robot-assisted surgical motions.
        in: Duncan J.S. Gerig G. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2005. Springer Berlin Heidelberg, 2005: 802-810
        • Rosen J.
        • Solazzo M.
        • Hannaford B.
        • Sinanan M.
        Task decomposition of laparoscopic surgery for objective evaluation of surgical residents’ learning curve using hidden markov model.
        Comput Aided Surg. 2002; 7: 49-61
        • Reiley C.E.
        • Lin H.C.
        • Varadarajan B.
        • et al.
        Automatic recognition of surgical motions using statistical modeling for capturing variability.
        Stud Health Technol Inform. 2008; 132: 396-401
        • Reiley C.E.
        • Hager G.D.
        Task versus subtask surgical skill evaluation of robotic minimally invasive surgery.
        in: Yang G.Z. Hawkes D. Rueckert D. Noble A. Taylor C. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2009. Springer Berlin Heidelberg, 2009: 435-442
        • Chen J.
        • Oh P.J.
        • Cheng N.
        • et al.
        Use of automated performance metrics to measure surgeon performance during robotic vesicourethral anastomosis and methodical development of a training tutorial.
        J Urol. 2018; 200: 895-902
        • Luongo F.
        • Hakim R.
        • Nguyen J.H.
        • Anandkumar A.
        • Hung A.J.
        Deep learning-based computer vision to recognize and classify suturing gestures in robot-assisted surgery.
        Surgery. 2020; 169: 1240-1244
        • Bromley M.
        • Marrou W.
        • Charles-de-Sa L.
        Evaluation of the number of progressive tension sutures needed to prevent seroma in abdominoplasty with drains: a single-blind, prospective, comparative, randomized clinical trial.
        Aesthet Plast Surg. 2018; 42: 1600-1608