Development and Feasibility Testing of a Device Briefing Tool and Training to Improve Patient Safety During Introduction of New Devices in Operating Rooms: Best Practices and Lessons Learned

Our team included four members from Ariadne Labs (two academic surgeon-researchers, one surgery resident, and one PhD public health senior program manager—W.B., J.H., J.L., and R.S.) and two physicians (C.L. and S.P.) from the J&J Medical Safety Organization. The Medical Safety Organization is an independent organization within J&J which is free of potential commercial or development portfolio bias. It provides independent and objective decision-making and accountability on patient safety issues.

In Thailand, we identified six hospitals to serve as pilot sites. Within these hospitals, J&J and Ariadne staff identified teams of surgeons and nurses to participate in the pilot study. The goal was to create a multidisciplinary training opportunity where J&J representatives could train surgeons and nurses together. Surgical technicians do not appear to be significantly utilized in Thai operating rooms and thus were not included in this training. Thailand was chosen as a study site given significant pre-established collaborations between J&J and several Thai hospitals focused on improving patient safety. Given the short timeline of this study, having pre-established stakeholder support and buy-in greatly facilitated the efficiency and efficacy of our work.

The Thai pilot hospitals were all teaching hospitals and considered tertiary referral centers. There was a mix of specialties represented: ranging from general surgery, hepatobiliary pancreas, surgical oncology, colorectal, and transplant. The hospitals ranged in size from 500 to 2000 inpatient beds.

The representatives who served as the key trainers during the multidisciplinary trainings on the use of the DBT all had university degrees (common areas of concentration: life science, medical technology). They understood English but were not fluent speakers. There was a variable range of time of employment at J&J from 1 to 10 y.

The creation of our DBT and training materials was a multidisciplinary and iterative process. This process involved generation of items for inclusion by expert consensus followed by review and feedback by practicing surgeons and nonclinician safety experts. A revised 4-item version of the DBT was then shared with J&J and additional practicing surgeons (n = 5) outside of Ariadne Labs. The additional practicing surgeons all practice in the United States and come from the following specialties: General Surgery, Trauma Surgery, and Surgical Critical Care. This was a convenience sample of surgeons well versed in quality improvement and global health and known by the principal investigator and another member of the research team. Overall, the key feedback from this rapid cycle feedback process included the following: 1) the DBT should be kept short and manageable, given the significant time constraints within the operating room. 2) The DBT should include the following: recognition of the moment when a new device will be used, confirmation that the J&J Optimized Device Performance guide was reviewed, and inclusion of a statement by the surgeon to ensure that anyone with concerns or questions speaks up. The standard and pictorial versions of the DBT are available in the Appendix. The Optimized Device Performance guide is a resource tool developed and disseminated by the Johnson & Johnson Medical Device Companies which is intended to help surgeons use their new device in accordance with the Instructions for Use. The Optimized Device Performance guide and Instructions for Use provide information on the safe and efficacious use of the device. It identifies key procedural risks, warnings and precautions, and appropriate risk mitigation strategies, again intended to help surgeons to achieve optimized device performance and procedural outcomes. For this collaboration, the Optimized Device Performance guide was specific to the sealing device that was being launched.

Although the DBT was designed to be used for any device, the device used in the pilot study was new surgical large jaw tissue sealer intended for open surgical procedures where vessel ligation and division is needed.

In addition to the creation of the DBT, the Ariadne team also developed a package of training materials that could be used for future implementation to address the needs of both surgeons and nurses. The package includes a standardized curriculum and accompanying presentations for training of the representatives and for representatives to use when training clinicians. The Ariadne Team also created a DBT training video that provides an overview of the Ariadne/J&J collaboration, a primer on the Surgical Safety Checklist and importance of patient safety in surgical care, an introduction to the DBT and the Optimized Device Performance guide, and a mock simulation of the DBT in a high fidelity simulated operating room with a human patient simulator. Feedback was continuously incorporated from our J&J collaborators and their Thai Patient Safety coordinator (a native Thai speaker) was able to provide Thai subtitles.

The training curriculum presentation and video were then used by our principal investigator Dr Joaquim Havens (a practicing surgeon and researcher) to train all 10 representatives using a web-based training. Dr Havens does not speak Thai and so a local J&J collaborator, based in Bangkok, translated the training. Subsequently, these 10 representatives used our training curriculum presentation and video to train 15 surgeons and 30 nurses over the course of 16 team trainings before their use of the DBT in the operating room. The surgeon specialties represented included General Surgery, Transplant Surgery, Vascular Surgery, and Surgical Oncology. All nurses were operating room nurses. While we did not record exact dates of the trainings and the first time of use in the operating room, the overall period was approximately 4 mo, between January 2018 and April 2018. We also worked to ensure that we timed the trainings to take place as close to the first use as possible. Thus, all trainings occurred the day before or on the day of first use in the operating room.

To assess our intervention (of the DBT and multidisciplinary trainings), we administered surveys to all representatives and clinicians participating in the DBT trainings. All surveys are available in the Appendix. The goals of the surveys were to understand the perceptions of representatives and clinicians regarding the training process and capture feedback from representatives and clinicians that could be used to improve the DBT. Surveys were given to representatives at the baseline, after representatives training by Ariadne Labs team, after representatives' training of clinicians, and after representatives’ observation of clinicians using the DBT in the operating room. Survey was given to clinicians at the baseline (before training), after training, and after the first operation when they used the DBT. The overall period across which the surveys were administered was from January 2018 to April 2018 and covered 15 cases (wherein one operation involved two surgeons).

The methodology for survey development included creation of key domains to be covered in each of the surveys. Domains included evaluation of current training; attitudes on Optimized Device Performance guide; knowledge of, familiarity with, and attitudes about Surgical Safety Checklist; attitudes on patient safety; and feedback on proposed training model and DBT. We then drafted questions for each of the domains and used Likert scales for most questions. Feedback on survey questions was obtained through semistructured interviews and focus groups. Four rounds of feedback were used to generate the final surveys that captured feedback from surgeons (n = 8), public health experts (n = 5), and a qualitative research specialist and research methodologist.

We report simple counts and frequencies of survey responses. Because the sample of respondents is small, with low statistical power, and not representative of a larger clinical population, we did not conduct inferential statistics or significance tests. All descriptive statistics were calculated using SAS 9.4.

The study was approved by Institutional Review Board at the Office of Human Research Administration at the Harvard T.H. Chan School of Public Health.

Eleven representatives were trained and 10 representatives completed all surveys (baseline, post-rep training, postclinician [surgeon and nurse] training, and postsurgery). Sixteen surgeons participated, 15 completed the baseline and postclinician training surveys, and 14 completed the postsurgery survey. Thirty nurses participated, 30 completed the baseline survey, and 29 completed the postclinician training and postsurgery surveys.

Overall, 90% all participants reported that the goals of the collaboration were clear through the training, the training principles and objectives were clear, and the trainings very much or completely met all learning objectives. However, when asked “following training, how prepared do you feel to train clinicians using the DBT?,” three representatives felt very prepared, two felt prepared, and half (or n = 5) felt only slightly prepared.

Clinicians were asked about the trainings regarding new devices they received in the past and what their perceptions were regarding the usefulness of these past trainings. The data are captured in Table 1 and show that many formats were used in prior new device trainings and simulations are perceived to be the most useful by clinicians. After our trainings, clinician confidence in using new devices rose (as seen in Table 2). There was no notable difference in clinician confidence based on being trained by representatives that felt “prepared to train clinicians” versus those that did not.

Finally, over 90% of clinicians feel confident about using the DBT, are very likely to use the DBT the next time they use a new device, and want the DBT used if they have surgery. Over 80% of clinicians felt that the length of the DBT was correct, the time to complete the DBT was correct, and the DBT fit well with their workflow. In terms of timing to use the DBT, 64% of clinicians want to use the DBT right before entering the operating room and 27% want to use the DBT in the operating room before starting their case.

Baseline familiarity with the Surgical Safety Checklist varied across representatives and clinicians but increased after training (Table 3). Representatives appeared to have the lowest familiarity with the SURGICAL SAFETY CHECKLIST at the baseline. In addition, there were some participants that stated they had never heard of the Surgical Safety Checklist, even after the training. The biggest barriers to Surgical Safety Checklist use among clinicians are captured in Table 4. Overall, the lack of perceived need and the belief that the Surgical Safety Checklist was too time-intensive were most frequent complaints. Of note, after training, the number of nurses expressing a lack of perceived need regarding the Surgical Safety Checklist dropped from 41% (n = 19) at the baseline to 10% (n = 22) after training. Despite variable familiarity with and perceptions of the Surgical Safety Checklist over 90% of clinicians and representatives would want the Surgical Safety Checklist used if they themselves were having surgery.

The data are captured in Table 5 and show that a clear majority of clinicians felt that the single most important reason to use the DBT was that it “makes the operating room team aware that a new device is being used.” Overall, clinicians expressed their support for the DBT, stating that it was easy to use, easy to understand, contained helpful information, better prepared the team to use a new device, promoted more confidence in the operating room, and ensured a shared understanding between surgeons and nurses.

In this feasibility and acceptability trial, we leveraged the established role and access of the sales representatives to trains surgeons and nurses. Other studies might employ different types of trainers. While representatives in our study were largely very positive about the training they received, half did not feel well prepared to train clinicians. Furthermore, our Ariadne Labs team conducted six onsite observations of the representatives while they trained clinician groups. The representatives were professional, highly prepared, and engaged well with both surgeons and nurses. However, from our survey results, it was notable that only 50% of representatives stated that they felt prepared to train clinicians, although the vast majority of clinicians found the trainings by the representatives to increase their confidence in using the DBT. If we were to speculate about the etiology of this, it was notable that the representatives comfort level appeared to be much greater when having one-on-one sessions with the clinicians versus when presenting to large groups. And thus, their perceived lack of preparedness might be reflective of a lack of confidence with training large groups and/or public speaking. Given that the representatives are well respected, able to function with significant autonomy, and have access to large numbers of surgeons and nurses; it appears that continuing to use them as trainers has significant potential. However, to further improve this delivery model, trainings might need to be adapted to further emphasize patient safety (versus technical device safety alone) and instruction for representatives that specifically focuses on group teaching/public speaking. Overall, while representatives have been focused on providing training regarding the technical aspects of a new device, our intervention started to explore the possibilities of expanding their role to also promoting safety and culture and improvement. Of note, there was no formal auditing of the training provided to the surgeons and nurses by the representatives other than the six observations. However, one of our JNJ collaborators was present for all trainings to answer questions and provide general quality control. In the future, a more formal auditing process could potentially yield more information to better gauge the needs of surgeons and nurses and further customize/improve future trainings.

Our onsite observations provided the opportunity to reflect and develop certain key best practices that could inform future roll-outs of briefing tools and multidisciplinary trainings. The proposed best practices are summarized in the following:

• •
  Provide options for trainings: surgeons and nurses separately or together.
• •
  Ensure training on DBT near the time of use in an operation.
• •
  Designate time for debriefings following all training sessions (of representatives and clinicians) to focus on what went well and what can be improved for future sessions.
• •
  Be open to ongoing development to adapt the tool and training based on feedback from participants.
• •
  Promote local customization of the tool and training to best meet local needs.

Our study has several limitations. First, given constraints of time and resources, we were only able to measure acceptability and feasibility of our training and device briefing tool, with this study population versus any other patient or clinical outcomes. Second, given the small sample sizes of representatives and clinicians, our data analysis is strictly descriptive in nature with no inferential statistical analysis. Third, while Table 2 shows surgeon's perceptions of their confidence with the new devices before and after use of the DBT, we did not perform any audits of the surgeon's actual familiarity and frequency of use with the sealing device before implementation of the DBT. Varying levels of familiarity with the device from prior use, research, or continuing medical education might have influenced surgeon/nurse attitudes toward the necessity of DBT use. Of note, this tool is meant to be used when teams are not familiar with the new device. It's important to use the DBT if even one surgical team member is new to using the device, but if all are familiar, we foresee that the tool is less necessary. Fourth, the survey questions asking about the use of the Surgical Safety Checklist in Thai operating rooms likely underestimated both understanding and use because we did not fully understand the local context. Specifically, our survey data showed low to medium awareness of the Surgical Safety Checklist before and after our training. This is in line with past data that while Surgical Safety Checklists have been shown to reduce morbidity and mortality worldwide, their implementation is highly variable.^{,  ,  ,}  However, onsite observations and discussions with representatives and clinicians at many of the Thai hospitals provided further context on this issue. Representatives and clinicians noted that while many seem to be unfamiliar with the Surgical Safety Checklist terminology, they implement a “timeout” in the operating room routinely, which appears to include key elements of the Surgical Safety Checklist and could be helpful in promoting speaking up and teamwork. Thus, in the future, while there is always room for improved Surgical Safety Checklist implementation; there appears to also be the need for greater adaptation of our survey questions to better match local language and context. Fifth, there was a significant pre-established collaboration between J&J and the selected hospitals that focused on improving patient safety culture. This pre-established relationship could have affected the results of the present study and there were no specific ways that our research team attempted to control for that influence.