Advertisement

General Surgery Residency Virtual Recruitment During the Pandemic: An Analysis of Applicant Surveys

Published:November 08, 2022DOI:https://doi.org/10.1016/j.jss.2022.10.015

      Abstract

      Introduction

      The COVID-19 pandemic forced a sudden change from in-person to virtual interviews for the general surgery residency match. General surgery programs and applicants adopted multiple strategies to best mimic in-person recruitment. The purpose of this study was to evaluate applicant opinions of the virtual recruitment format.

      Materials and methods

      Postinterview survey responses for applicants interviewing at a single general surgery residency program in the 2020-2021 and 2021-2022 cycles were evaluated. All interviewed applicants were sent an anonymous survey assessing the virtual interview structure, their impression of the program, and their opinions on recruitment in the future.

      Results

      The response rate was 31.2% (n = 60). Most (88.4%) respondents reported a more favorable view of the program after a virtual interview. Factors that were most likely to create a favorable impression were residents (89.6%) and culture (81.0%). 50.8% of applicants favored virtual-only interviews. The majority of applicants (60.3%), however, preferred the virtual interview remain a component of the application process, 34.4% recommended that virtual interviews be used as an initial screen before in-person invites, while 19.0% suggested applicants should interview in-person or virtually without penalty. 62.1% favored capping the number of interviews offered by programs and accepted by applicants.

      Conclusions

      The virtual interview format for general surgery residency allows applicants to effectively evaluate a residency program. Applicants are in favor of a combination of virtual and in-person interviews in the future. Innovation in the recruitment process, including limiting the number of applications and incorporating virtual events, is supported by applicants.

      Keywords

      Introduction

      The residency interview process is a competitive, complex, expensive, but integral component of medical education in the United States.
      • Pourmand A.
      • Lee H.
      • Fair M.
      • Maloney K.
      • Caggiula A.
      Feasibility and usability of tele-interview for medical residency interview.
      ,
      • Wolff M.
      • Burrows H.
      Planning for virtual interviews: residency recruitment during a pandemic.
      Traditionally, in-person interviews offered applicants an opportunity to see an institution firsthand, interact with both residents and faculty, and explore different cities. In addition to academics, factors such as morale, collegiality, and resident happiness play an important part in applicants' rankings of programs; many applicants consider these to be best conveyed in person.
      • Pourmand A.
      • Lee H.
      • Fair M.
      • Maloney K.
      • Caggiula A.
      Feasibility and usability of tele-interview for medical residency interview.
      ,
      • Azizzadeh A.
      • McCollum C.H.
      • Miller C.C.
      • Holliday K.M.
      • Shilstone H.C.
      • Lucci A.
      Factors influencing career choice among medical students interested in surgery.
      • Kim R.H.
      • Gilbert T.
      • Suh S.
      • Miller J.K.
      • Eggerstedt J.M.
      General surgery residency interviews: are we following best practices?.
      • Mayer K.L.
      • Perez R.V.
      • Ho H.S.
      Factors affecting choice of surgical residency training program.
      • Sinno S.
      • Mehta K.
      • Squitieri L.
      • et al.
      Residency characteristics that matter most to plastic surgery applicants: a multi-institutional analysis and review of the literature.
      • Zuo K.J.
      • Retrouvey H.
      • Wanzel K.R.
      Factors that affect medical students' perception and impression of a plastic surgery program: the role of elective rotations and interviews.
      • Labiner H.E.
      • Anderson C.E.
      • Maloney Patel N.
      Virtual recruitment in surgical residency programs.
      In-person interviews have also been valued by program directors as a tool to evaluate applicants’ personal characteristics and fit within the program.
      • Pourmand A.
      • Lee H.
      • Fair M.
      • Maloney K.
      • Caggiula A.
      Feasibility and usability of tele-interview for medical residency interview.
      ,
      • Kim R.H.
      • Gilbert T.
      • Suh S.
      • Miller J.K.
      • Eggerstedt J.M.
      General surgery residency interviews: are we following best practices?.
      The COVID-19 pandemic necessitated sudden, sweeping changes in all aspects of medicine, including graduate medical education recruitment.
      • Ammann A.M.
      • Cortez A.R.
      • Vaysburg D.M.
      • et al.
      Examining the impact of COVID-19 restrictions on the operative volumes of US general surgery residents.
      ,
      • Van Haren R.M.
      • Delman A.M.
      • Turner K.M.
      • et al.
      Impact of the COVID-19 pandemic on lung cancer screening program and subsequent lung cancer.
      In light of this, the Accreditation Council of Graduate Medical Education and the American Medical Association committed to a virtual interview format for the entire 2020-2021 residency interview cycle, and most programs continued to offer virtual-only interviews through the 2021-2022 cycle.
      • Wolff M.
      • Burrows H.
      Planning for virtual interviews: residency recruitment during a pandemic.
      ,
      • Zaki M.M.
      • Nahed B.V.
      Utilizing virtual interviews in residency selection beyond COVID-19.
      Residency and fellowship programs across all specialties developed several different strategies to shift their interview process to a virtual platform, yet it is unknown how effective these were compared to the in-person format.
      • Wolff M.
      • Burrows H.
      Planning for virtual interviews: residency recruitment during a pandemic.
      ,
      • Zaki M.M.
      • Nahed B.V.
      Utilizing virtual interviews in residency selection beyond COVID-19.
      • Duque S.
      • Riccelli V.
      • Mulqueen S.
      • Zhang A.Y.
      Global pandemic and PRS residency match: can social media fill the void?.
      • Majumder A.
      • Eckhouse S.R.
      • Brunt L.M.
      • et al.
      Initial experience with a virtual platform for advanced gastrointestinal minimally invasive surgery fellowship interviews.
      • Nguyen J.K.
      • Shah N.
      • Heitkamp D.E.
      • Gupta Y.
      COVID-19 and the radiology match: a residency program's survival guide to the virtual interview season.
      • Taylor M.
      • Wallen T.
      • Mehaffey J.H.
      • et al.
      Interviews during the pandemic: a thoracic education cooperative group and surgery residents project.
      • Vining C.C.
      • Eng O.S.
      • Hogg M.E.
      • et al.
      Virtual surgical fellowship recruitment during COVID-19 and its implications for resident/fellow recruitment in the future.
      Our objective was to determine if general surgery residency applicants thought that they could successfully evaluate a program through a virtual interview and to assess their opinion on interview formats moving forward. Given the significant time and financial burden of in-person interviews, these findings may help to optimize the residency interview process and inform program directors how to integrate novel recruitment strategies in the future.

      Materials and Methods

      Surveys were distributed by email to all general surgery applicants who interviewed at a single urban Midwestern program after their respective interview days. In the years 2015, 2016, and 2018, a survey was distributed after each applicant's interview day with the goal of using responses to improve the interview experience. Survey questions were answered with a Likert scale (Poor, Fair, Good, Very Good, and Outstanding). A separate survey was developed to evaluate recruitment effectiveness of the virtual format at a single institution for the 2020-2021 and 2021-2022 general surgery residency interview seasons. This survey was constructed using prior literature on virtual interviews in residents and fellows.
      • Pourmand A.
      • Lee H.
      • Fair M.
      • Maloney K.
      • Caggiula A.
      Feasibility and usability of tele-interview for medical residency interview.
      ,
      • Labiner H.E.
      • Anderson C.E.
      • Maloney Patel N.
      Virtual recruitment in surgical residency programs.
      ,
      • Zaki M.M.
      • Nahed B.V.
      Utilizing virtual interviews in residency selection beyond COVID-19.
      ,
      • Majumder A.
      • Eckhouse S.R.
      • Brunt L.M.
      • et al.
      Initial experience with a virtual platform for advanced gastrointestinal minimally invasive surgery fellowship interviews.
      ,
      • Vining C.C.
      • Eng O.S.
      • Hogg M.E.
      • et al.
      Virtual surgical fellowship recruitment during COVID-19 and its implications for resident/fellow recruitment in the future.
      • D'Angelo J.D.
      • D'Angelo A.L.D.
      • Mathis K.L.
      • Dozois E.J.
      • Kelley S.R.
      Program director opinions of virtual interviews: whatever makes my partners happy.
      • Daram S.R.
      • Wu R.
      • Tang S.J.
      Interview from anywhere: feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process.
      • Edje L.
      • Miller C.
      • Kiefer J.
      • Oram D.
      Using skype as an alternative for residency selection interviews.
      • Grova M.M.
      • Donohue S.J.
      • Meyers M.O.
      • Kim H.J.
      • Ollila D.W.
      Direct comparison of in-person versus virtual interviews for complex general surgical oncology fellowship in the COVID-19 era.
      • Healy W.L.
      • Bedair H.
      Videoconference interviews for an adult reconstruction fellowship: lessons learned.
      • Lewit R.
      • Gosain A.
      Virtual interviews may fall short for pediatric surgery fellowships: lessons learned from COVID-19/SARS-CoV-2.
      • Lund S.
      • MacArthur T.
      • Keune J.
      • et al.
      Showcasing a general surgery residency program during the COVID-19 pandemic.
      • Phillips B.T.
      • Gosman A.A.
      • Maricevich R.S.
      • Rodriguez E.D.
      • Rohrich R.J.
      The plastic surgery residency interview revisited: virtual interviews and beyond.
      • Rajesh A.
      • Asaad M.
      • Elmorsi R.
      • Ferry A.M.
      • Maricevich R.S.
      The virtual interview experience for match 2021: a pilot survey of general surgery residency program directors.
      Surveys were voluntary and anonymous, and recruitment emails stated that participation in the survey implied consent to participation in the study. No reimbursement was offered and assurances were made that participation and opinions expressed in the survey did not affect match rankings. The research was approved by our institutional review board: 2021-0442.

      Description of the interview process

      Prior to the 2020-2021 interview season, the program's website was updated with professionally recorded informational videos that included interviews of both residents and faculty. The interview season began with a series of three voluntary “virtual meet-and-greet” sessions for applicants to learn about the program prior to submitting their Electronic Residency Application Service application. These were advertised on the program's social media accounts and were attended by residents using Microsoft Teams. Applicants and residents met in multiple breakout rooms to facilitate small-group conversations. This event was not continued in the 2021-2022 season, as the prior season's survey indicated that the majority of respondents did not attend.
      In both interview seasons, a holistic Electronic Residency Application Service application review was performed. Approximately 16 applicants were interviewed on each of the 6 different virtual interview dates between November and January. Prior to each session, a care package was mailed to applicants containing information about our program, as well as a collection of items representing the program and city. Applicants were invited to a virtual “meet-the-residents” event the night prior to their interview. Each session started with an introduction from the program director, including a video tour of the hospital. This was followed by informal breakout sessions of three to four residents and three to four applicants per room, rotating every 15 min to maximize applicants and resident interaction.
      On the interview day, all applicants attended our virtual morbidity and mortality conference, followed by information sessions held by the administrative chief resident and the program directors. The group of applicants was then divided in half for a morning and afternoon interview session. Each applicant participated in six to eight 15-min faculty interviews, one of which was with the department chair and program director. Between interviews, applicants joined virtual “breakroom” sessions where they were able to interact with a group of residents.
      At the conclusion of each interview day, an anonymous, voluntary survey was emailed to all applicants. The survey for the 2020-2021 cycle consisted of 22 total questions in both multiple choice and text-box format, with the option to comment on any question (Table 1, Table 2). The 2021-2022 survey excluded two questions: one regarding the “virtual meet-and-greet” sessions that had not been held that year, and another regarding “signaling” programs that the applicants were most interested in, an option that became available for the 2021-2022 cycle. The survey did not acquire demographic information to preserve the anonymity of respondents.
      Table 1Pre-2020 postinterview survey questions.
      QuestionOutstandingVery goodGoodFairPoorTotal
      n%N%n%n%n%N
      Welcome reception on Tuesday evening7070.00%3030.00%00.00%00.00%00.00%100
      Interview process on Wednesday4444.00%5151.00%44.00%11.00%00.00%100
      Surgery resident workspace/Lounge2929.00%5252.00%1717.00%22.00%00.00%100
      Reputation of the department5353.00%4141.00%66.00%00.00%00.00%100
      Quality of current residents7272.00%2727.00%11.00%00.00%00.00%100
      Resident morale6262.00%3232.00%66.00%00.00%00.00%100
      Resident operative experience5555.00%3636.00%88.00%11.00%00.00%100
      Research opportunities5959.00%2929.00%1010.00%22.00%00.00%100
      Geographic location2020.00%4040.00%2626.00%1313.00%11.00%100
      Post residency placement of chief residents8383.00%1515.00%22.00%00.00%00.00%100
      Table 2Virtual postinterview survey results.
      QuestionResponses
      1. If you had to travel for interviews this year, would you still have interviewed at [this program]?60
       Yes59 (98.3%)
       No1 (1.7%)
      2. Did your visit and interviews leave you with a more or less favorable view of our training program than you had preconceived before interviewing?60
       Much less favorable1 (1.7%)
       Less favorable3 (5.0%)
       No difference3 (5.0%)
       More favorable25 (41.7%)
       Much more favorable28 (46.7%)
      3. Was your final impression of our program favorable compared to others?60
       Much less favorable1 (1.7%)
       Less favorable1 (1.7%)
       No difference2 (3.3%)
       More favorable29 (48.3%)
       Much more favorable27 (45.0%)
      4. If not favorable, why?15
       Geography5 (33.3%)
       Teaching faculty2 (13.3%)
       Education faculty1 (6.7%)
       Residents1 (6.7%)
       Research opportunities0 (0%)
       Operative experience1 (6.7%)
       Other scholarly opportunities1 (6.7%)
       Culture2 (13.3%)
       Wellness0 (0%)
       Pay/benefits1 (6.7%)
       Other6 (40.0%)
      5. If favorable, why?58
       Geography25 (43.8%)
       Teaching faculty45 (77.5%)
       Education faculty36 (62.0%)
       Residents52 (89.6%)
       Research opportunities47 (81.0%)
       Operative experience45 (77.5%)
       Other scholarly opportunities17 (29.3%)
       Culture47 (81.0%)
       Wellness17 (29.3%)
       Pay/benefits2 (3.4%)
       Other1 (1.7%)
      6. Did the previsit care-package help shape your impression of our program?60
       Much less favorable0 (0%)
       Less favorable0 (0%)
       No difference8 (13.3%)
       More favorable31 (51.7%)
       Much more favorable21 (35.0%)
      7. How did the Tuesday evening “meet-the-residents” event compare to others you attended?60
       Much less favorable0 (0%)
       Less favorable3 (5.0%)
       No difference19 (31.7%)
       More favorable26 (43.3%)
       Much more favorable12 (20.0%)
      8. Was attending the morbidity and mortality conference worthwhile?60
       Yes51 (85.0%)
       No9 (15.0%)
      9. How were the interview-day informational sessions compared to others you have attended?60
       Much less favorable0 (0.00%)
       Less favorable1 (1.7%)
       No difference23 (38.3%)
       More favorable30 (50.0%)
       Much more favorable6 (10.0%)
      10. How were the interview sessions compared to others you have attended?60
       Much less favorable0 (0%)
       Less favorable3 (5.0%)
       No difference14 (23.3%)
       More favorable34 (56.6%)
       Much more favorable9 (15.0%)
      11. Would you prefer fewer or more interviews with our faculty members?60
       Fewer interviews7 (11.7%)
       About right number50 (83.3%)
       More interviews0 (0%)
       No more interviews but longer time per interview.3 (5.0%)
      12. Would you prefer more or less time getting to know our residents?60
       Much less time0 (0%)
       Less time1 (1.7%)
       Same time46 (76.6%)
       More time13 (21.6%)
       Much more time0 (0%)
      13. How was the website compared to others you have visited?60
       Much less favorable1 (1.7%)
       Less favorable3 (5.0%)
       No difference23 (38.3%)
       More favorable28 (46.7%)
       Much more favorable5 (8.3%)
      14. How were any pre or postvisit informational sessions you may have attended compared to others you attended? Example: Open houses, diversity virtual second visit… (question only appeared on the 2020-2021 survey)27
       Much less favorable0 (0%)
       Less favorable0 (0%)
       No difference9 (33.3%)
       More favorable3 (11.1%)
       Much more favorable1 (3.7%)
       Did not attend14 (51.8%)
      15. What additional information did other programs discuss that we missed or that could have helped with your decision making?-
       Open ended response-
      16. If more robust program information were available beforehand, would that have allowed you to reduce the number of programs you interviewed for?60
       Yes15 (25.0%)
       No15 (25.0%)
       Would still have interviewed at all programs that invited me.30 (50.0%)
      17. If programs capped the number of interviews they offered, would you be in favor of also capping the number of interviews an applicant was allowed to accept? For example, programs limited to 10 interviews per categorical slot, applicants limited to 10 program interviews58
       Yes36 (62.1%)
       No22 (37.9%)
      18. If you had to select 5 programs to “signal” or indicate significant interest to before interview offers are sent out (as was done in otolaryngology this year), would [this program] have made your top 5? (Question only appeared on the 2020-2021 survey)28
       Yes15 (53.6%)
       No13 (46.4%)
      19. Did we rank in your top 5 after you interviewed with us?57
       Yes42 (73.7%)
       No15 (23.3%)
      20. Do you recommend virtual only interviews for next year?57
       Yes29 (50.8%)
       No28 (49.1%)
      21. Do you recommend keeping virtual interviews only as a component of the application process for next year?58
       Yes, but only as an informal informational part of the process4 (6.9%)
       Yes, applicants can interview in-person or virtual without penalty11 (19.0%)
       Yes, should use virtual interviews as an initial screen before an in-person interview20 (34.4%)
       No, just unneeded if in-person is required eventually23 (39.7%)
      22. Other suggestions to make overall matching process more effective and efficient:-
       Open ended response-
      Six mo into their postgraduate year (PGY)-1 year, the six residents who matched in the 2020-2021 cycle were sent an anonymous one-question survey regarding the accuracy of this program's representation through virtual interviews. Another anonymous, 1-question survey regarding applicants' abilities to represent themselves virtually was sent to all faculty interviewers at the conclusion of the 2021-2022 season. Response data for all surveys were summarized using descriptive statistics.
      Finally, the cost of conducting the interview season for the 2020-2021 season was calculated and compared to the average cost of three prior interview seasons. This study was reviewed and approved by the University of Cincinnati Institutional Review Board.

      Results

      Previrtual interview season

      The survey administered in the years 2015, 2016, and 2018 had 100 of 221 applicants responded (45.24%). Aspects of the program most frequently ranked “Outstanding” were “Postresidency Placement of Chiefs,” “Quality of Current Residents,” “Welcome Reception,” “Resident Morale,” “Research Opportunities,” and “Operative Experience.”

      Evaluation of the program

      Ninety-seven total applicants were interviewed in the 2020-2021 cycle (28 responses), and 96 in the 2021-2022 cycle (32 responses) with an overall response rate of 31.2%. (Table 1, Table 2). A majority of applicants (88.4%) responded that the interview gave them a more favorable or much more favorable view of the program than they had preconceived. Most applicants (93.3%) also replied that their final impression of the program was more favorable or much more favorable compared to other programs. Factors that most frequently influenced a favorable impression were: residents (89.6%), culture (81.0%), research opportunities (81.0%), teaching Faculty (77.5%), and operative experience (77.5%) (Fig. 1). Multiple open-ended comments positively mentioned residents and culture as factors that helped applicants form their opinion (Table 3). Applicants also viewed the preinterview evening “meet-the-residents” events (63.3%), interview day informational sessions (60.0%), and interview sessions (71.6%), more favorably or much more favorably compared to those of other programs.
      Figure thumbnail gr1
      Fig. 1Factors associated with a favorable impression of the program.
      Table 3Representative quotes from postinterview survey results.
      5. If [your final impression of our program compared to others was] favorable, why?
       CULTURE CULTURE CULTURE!!!! the resident social was hands down the best social I attended through the entire interview season.
       There was a genuine culture of kindness and support between residents and faculty.
      17. If programs capped the number of interviews they offered, would you be in favor of also capping the number of interviews an applicant was allowed to accept? For example, programs limited to 10 interviews per categorical slot, applicants limited to 10 program interviews
       I think I would prefer a cap on the number of schools I could apply to rather than the number of interviews I can attend.
       I think this may be a good idea, as long as there are certain considerations for special cases, such as couple matches who often have to interview at more programs.
       Couples matching so would not want interview cap in fear that we wouldn't match together
       I think 10 is far too little. I Think a cap at 16 would be reasonable
       This is tough because it is difficult for an applicant to gauge their interest in a program before interviewing. If an applicant were to be capped at 10 and they only liked 6 of the programs when they could have had the opportunity to interview at programs that would be a better fit, it would result in a lot of disappointment. A better alternative would be to cap the number of programs an applicant to apply to.
      20. Do you recommend virtual only interviews for next year?
       Mixed format would be best.
       With In-person second looks.
       Virtual only interviews with options for fewer in-person visits.
       With second look in-person.
       Would need to be even among all applicants (would not have virtually interviewed if others were able to visit in person) interviews virtually and second looks in-person would be great.
       Perhaps for preliminary 1-2 y positions, but not for categorical spots.
       I would recommend an interview structure that has two parts: a 1st round of virtual interviews with an in-person 2nd round (or 2nd look) of interviews.
       With the option to visit the campus if the applicant is able to.
       It would be nice to have two-tier interviewing. Short virtual interviews with in-person interviews with your top couple of programs.
       I appreciate the money I saved but would never have traded that for getting to experience each program in-person.
       I find that this is an important step for mitigating the financial barrier that may keep medical students from traveling to multiple interviews. I Believe virtual only interviews helps with levelling the playing field and allowing people to attend interviews that they may otherwise not be able to attend.
       Absolutely not. Have had several in-person interviews this Gen Surg cycle and can attest that in person interviews are way better for applicants. Much better opportunities to observe resident and resident-attending interactions, get a feel of the city and hospital system, and really get a better glimpse of the program and the types of residents and also applicants it attracts. I Understand the cost and time advantage of virtual interviews but I think that being able to see and experience a program in person is well worth it and I would have 100% preferred in-person interviews over virtual.
      21. Q21 do you recommend keeping virtual interviews only as a component of the application process for next year?
       While not exactly the same as in-person, virtual interviews are much more accessible for applicants. I Think that they should be kept as an option, particularly for applicants from other time zones or with financial need.
       Can use in-person for second look.
       I think virtual should be some component. Perhaps a second-look that is in-person.
       If used wisely as a screening process for both programs and applicants I could see how this might be useful and help reduce the number of in-person interviews at little to no detriment to the applicants.
       I think using it as an initial screen would be a worthwhile idea.
      22. Other suggestions to make overall matching process more effective and efficient:
       I think there should also be an initial limit on the number of programs an applicant can apply to. After the first round or two of interview offers, this could open up for an additional round/rounds of applications. This would signal significant interest to the programs and potential help decrease the burden on applicants.

      Applicant opinions of future virtual recruitment

      Virtual-only interviews were preferred by 50.8% of applicants (Fig. 2A). However, when given the option of a virtual and in-person component, the majority of applicants (60.3%) were in favor (Fig. 2B). 34.4% of these applicants suggested that virtual interviews be used as an initial screen before an in-person interview, and 19.0% suggested that applicants could choose to interview either virtually or in-person without penalty. On the other hand, 39.7% stated that if an in-person interview is required, virtual interviews are unneeded. Written responses were also mixed, with multiple applicants supporting multitiered interviews and some commenting on the costs saved by virtual interviews (Table 3).
      Figure thumbnail gr2
      Fig. 2(A) Applicant opinions on virtual-only interviews. (B) Applicant opinions on virtual interviews as a component of recruitment.
      Most applicants (62.1%) were also in favor of capping the number of interviews that they were allowed to accept if programs capped the number of interviews offered (Table 2). Written responses to this question demonstrated concern, however, with some stating that this would be difficult for those who were matching with a significant other (couples matching) (Table 3). In the 2020-2021 survey, applicant opinions were split regarding signaling, and this question was removed for the following year's survey because signaling became an option.

      Opinions of this program's matched applicants and faculty interviewers

      The six applicants who matched at this program after the 2020-2021 season were asked, “How accurately did you think the virtual interview represented our program on a scale of 1-5? (1 being least accurately, 5 being most accurately),” 6 mo into their PGY-1 year. The response rate was 100%; four applicants responded “5”, while two responded “4,” with a mean of 4.8. The 13 faculty interviewers from both seasons were similarly asked, “How accurately did you think general surgery applicants were able to represent themselves through the virtual interview on a scale of 1-5? (1 being least accurately and 5 being most accurately).” Here, ten responded (76.9%), with a mean response of 3.7.

      Cost analysis

      The total cost of conducting interviews was calculated for the 2020-2021 virtual interview season, as well as for three prior in-person interview seasons. The average cost of the in-person interviews was $32,401.46 per season, compared to $23,258.67 in the virtual interview season, a savings of 28.2%. The cost of hosting in-person preinterview dinners as well as providing meals during the interview day accounted for 67% of spending for in-person interviews, compared to only 7% of virtual interview spending. Costs of recruitment materials, however, were increased for virtual interviews by inclusion of the care package sent to all applicants. The cost of these materials increased from an average of $10,640.03 per year for in-person interviews to $21,609.54 for the virtual interview.

      Discussion

      The sudden shift from in-person to virtual interviews presented a challenge for both programs and applicants in the general surgery match. The results of this single-institution postinterview survey suggest that applicants were able to adequately assess a program through a virtual interview. The majority of applicants favored virtual interviews as a component of future general surgery recruitment.
      Historically, the idea of virtual interviews has been met with hesitancy from both applicants and programs.
      • Pourmand A.
      • Lee H.
      • Fair M.
      • Maloney K.
      • Caggiula A.
      Feasibility and usability of tele-interview for medical residency interview.
      ,
      • Daram S.R.
      • Wu R.
      • Tang S.J.
      Interview from anywhere: feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process.
      Applicants’ rank lists are heavily influenced by interpersonal interactions with residents, perceptions of lifestyle, and program culture. These aspects of the interview are feared to be lost with the virtual format.
      • Azizzadeh A.
      • McCollum C.H.
      • Miller C.C.
      • Holliday K.M.
      • Shilstone H.C.
      • Lucci A.
      Factors influencing career choice among medical students interested in surgery.
      ,
      • Mayer K.L.
      • Perez R.V.
      • Ho H.S.
      Factors affecting choice of surgical residency training program.
      • Sinno S.
      • Mehta K.
      • Squitieri L.
      • et al.
      Residency characteristics that matter most to plastic surgery applicants: a multi-institutional analysis and review of the literature.
      • Zuo K.J.
      • Retrouvey H.
      • Wanzel K.R.
      Factors that affect medical students' perception and impression of a plastic surgery program: the role of elective rotations and interviews.
      ,
      • Duque S.
      • Riccelli V.
      • Mulqueen S.
      • Zhang A.Y.
      Global pandemic and PRS residency match: can social media fill the void?.
      ,
      • Taylor M.
      • Wallen T.
      • Mehaffey J.H.
      • et al.
      Interviews during the pandemic: a thoracic education cooperative group and surgery residents project.
      ,
      • Daram S.R.
      • Wu R.
      • Tang S.J.
      Interview from anywhere: feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process.
      ,
      • Lewit R.
      • Gosain A.
      Virtual interviews may fall short for pediatric surgery fellowships: lessons learned from COVID-19/SARS-CoV-2.
      ,
      • Leape L.L.
      • Brennan T.A.
      • Laird N.
      • et al.
      The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II.
      We addressed this in our interview structure with multiple breakout rooms to facilitate small-group conversations between residents and applicants. Subsequently, applicants chose “Residents” and “Culture” as reasons that influenced their favorable opinion, also referencing these factors in their open-ended responses. These responses, as well as favorable representation of research opportunities and operative experience in the virtual format are similar to those factors which applicants rated most highly after in-person interviews. Our results suggest that a virtual interview format is able to convey these aspects of an in-person interview and may serve an important role in deciding how to conduct interviews moving forward.
      Other research in both the general surgery residency and surgical fellowship matches has studied virtual interviews with mixed results. A survey found that virtual “webinars” gave residency applicants an adequate “feel” for their program but that they would still prefer in-person interviews.
      • Lund S.
      • MacArthur T.
      • Keune J.
      • et al.
      Showcasing a general surgery residency program during the COVID-19 pandemic.
      Residency program directors found that virtual interviews made the assessment of applicant fit more difficult, but would adopt a mixed format in the future.
      • Rajesh A.
      • Asaad M.
      • Elmorsi R.
      • Ferry A.M.
      • Maricevich R.S.
      The virtual interview experience for match 2021: a pilot survey of general surgery residency program directors.
      Some fellowship applicants preferred in-person interviews, but responded positively to the idea of virtual interviews being used as a screening tool in the future.
      • Taylor M.
      • Wallen T.
      • Mehaffey J.H.
      • et al.
      Interviews during the pandemic: a thoracic education cooperative group and surgery residents project.
      ,
      • Vining C.C.
      • Eng O.S.
      • Hogg M.E.
      • et al.
      Virtual surgical fellowship recruitment during COVID-19 and its implications for resident/fellow recruitment in the future.
      ,
      • Edje L.
      • Miller C.
      • Kiefer J.
      • Oram D.
      Using skype as an alternative for residency selection interviews.
      ,
      • Lewit R.
      • Gosain A.
      Virtual interviews may fall short for pediatric surgery fellowships: lessons learned from COVID-19/SARS-CoV-2.
      Other fellowship applicants and faculty felt that they were able to convey themselves well and evaluate each other adequately in the virtual format, preferring to keep it moving forward.
      • Vining C.C.
      • Eng O.S.
      • Hogg M.E.
      • et al.
      Virtual surgical fellowship recruitment during COVID-19 and its implications for resident/fellow recruitment in the future.
      ,
      • D'Angelo J.D.
      • D'Angelo A.L.D.
      • Mathis K.L.
      • Dozois E.J.
      • Kelley S.R.
      Program director opinions of virtual interviews: whatever makes my partners happy.
      ,
      • Leape L.L.
      • Brennan T.A.
      • Laird N.
      • et al.
      The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II.
      These studies, as well as ours, demonstrate that applicants feel comfortable evaluating training programs with a virtual platform, particularly as applicants matched to our institution felt that they had adequately assessed our program. With sufficient preparation, the virtual interview experience can leave both programs and applicants with similar impressions and information as an in-person interview.
      In 2015, the average cost of participating in in-person interviews across all specialties was $3422 per applicant, up to $6930 for some specialties, and was even higher for those who were couples matching.
      • Pourmand A.
      • Lee H.
      • Fair M.
      • Maloney K.
      • Caggiula A.
      Feasibility and usability of tele-interview for medical residency interview.
      Costs to programs include event spaces, transportation, and meals. Virtual interviews significantly lower this burden for both sides; in the 2020-2021 virtual interview season, our program's costs decreased by 28.2%, while surveys show applicants saved $500-$800 per interview.
      • Labiner H.E.
      • Anderson C.E.
      • Maloney Patel N.
      Virtual recruitment in surgical residency programs.
      In addition to the cost of interviewing, applicants spend a significant amount of time travelling, committing an average of 20 d to interviews.
      • Pourmand A.
      • Lee H.
      • Fair M.
      • Maloney K.
      • Caggiula A.
      Feasibility and usability of tele-interview for medical residency interview.
      ,
      • Edje L.
      • Miller C.
      • Kiefer J.
      • Oram D.
      Using skype as an alternative for residency selection interviews.
      Both interview costs and time spent away from education are higher for applicants travelling long distances, and travel can result in scheduling conflicts for back-to-back interviews. While some medical schools structure the year to afford travel time, and some applicants can afford the cost of interviews, others may have to limit their interviews based on cost alone, weakening their chances of matching.
      • Labiner H.E.
      • Anderson C.E.
      • Maloney Patel N.
      Virtual recruitment in surgical residency programs.
      ,
      • Walsh K.
      Why is medical education so expensive?.
      ,
      American Association of Medical Colleges, W.D
      AAMC diversity in medical education facts and figures: current trends in medical education.
      Some faculty fear that the ease of the virtual interview process would lead to candidates applying to many more programs than they would have otherwise, offsetting the balance of the process.
      • Labiner H.E.
      • Anderson C.E.
      • Maloney Patel N.
      Virtual recruitment in surgical residency programs.
      In the 2021-2022 cycle, the National Residency Matching Program implemented preference signaling, where applicants were able to notify their top 5 programs when submitting their initial application, with hopes that it would differentiate those applicants who had sincere interest in a program. This program had been successfully implemented in the otolaryngology match in the prior year, as applicants had high interview rates at their signaled programs and favored continuing preference signaling for future cycles.
      • Pletcher S.D.
      • Chang C.W.D.
      • Thorne M.C.
      • Malekzadeh S.
      The otolaryngology residency program preference signaling experience.
      Another possible strategy to mitigate this concern is to cap the number of in-person interviews that can be offered to or accepted by an applicant, a concept that has been supported by some specialties and was preferred by most of our survey respondents.
      • Salehi P.P.
      • Azizzadeh B.
      • Lee Y.H.
      Preference signaling for competitive residency programs in the NRMP.
      ,
      • Morgan H.K.
      • Winkel A.F.
      • Standiford T.
      • et al.
      The case for capping residency interviews.
      Given the demonstrated benefits to virtual interviews but mixed results on their effectiveness, we recommend considering a hybrid approach to the interview season. While applicants did not universally agree on continuing virtual-only interviews, the majority in our study and in others recommended keeping the virtual format as a part of the process.
      • Pourmand A.
      • Lee H.
      • Fair M.
      • Maloney K.
      • Caggiula A.
      Feasibility and usability of tele-interview for medical residency interview.
      ,
      • Zaki M.M.
      • Nahed B.V.
      Utilizing virtual interviews in residency selection beyond COVID-19.
      ,
      • Majumder A.
      • Eckhouse S.R.
      • Brunt L.M.
      • et al.
      Initial experience with a virtual platform for advanced gastrointestinal minimally invasive surgery fellowship interviews.
      ,
      • Taylor M.
      • Wallen T.
      • Mehaffey J.H.
      • et al.
      Interviews during the pandemic: a thoracic education cooperative group and surgery residents project.
      • Vining C.C.
      • Eng O.S.
      • Hogg M.E.
      • et al.
      Virtual surgical fellowship recruitment during COVID-19 and its implications for resident/fellow recruitment in the future.
      • D'Angelo J.D.
      • D'Angelo A.L.D.
      • Mathis K.L.
      • Dozois E.J.
      • Kelley S.R.
      Program director opinions of virtual interviews: whatever makes my partners happy.
      • Daram S.R.
      • Wu R.
      • Tang S.J.
      Interview from anywhere: feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process.
      • Edje L.
      • Miller C.
      • Kiefer J.
      • Oram D.
      Using skype as an alternative for residency selection interviews.
      ,
      • Healy W.L.
      • Bedair H.
      Videoconference interviews for an adult reconstruction fellowship: lessons learned.
      ,
      • Lund S.
      • MacArthur T.
      • Keune J.
      • et al.
      Showcasing a general surgery residency program during the COVID-19 pandemic.
      ,
      • Fogel H.A.
      • Liskutin T.E.
      • Wu K.
      • Nystrom L.
      • Martin B.
      • Schiff A.
      The economic burden of residency interviews on applicants.
      Looking back, our matched applicants and faculty believed that they had evaluated each other accurately as well. We believe that with the past 2 y of experience, programs can create virtual interview formats that are effective and sustainable. Program information, such as schedules, curriculum, and case volumes can be uploaded to a website, sent electronically, or presented as a video-lecture instead of being provided as a hard copy.
      • Pourmand A.
      • Lee H.
      • Fair M.
      • Maloney K.
      • Caggiula A.
      Feasibility and usability of tele-interview for medical residency interview.
      ,
      • Labiner H.E.
      • Anderson C.E.
      • Maloney Patel N.
      Virtual recruitment in surgical residency programs.
      ,
      • Lund S.
      • MacArthur T.
      • Keune J.
      • et al.
      Showcasing a general surgery residency program during the COVID-19 pandemic.
      ,
      • Rajesh A.
      • Asaad M.
      Alternative strategies for evaluating general surgery residency applicants and an interview limit for match 2021: an impending necessity.
      ,
      • Buonpane C.
      • Young S.
      • Tuliszewski R.
      • et al.
      Do's and don'ts of the virtual interview: perspectives from residency and fellowship applicants.
      Applicants have also responded positively to on-site hospital tours being substituted with prerecorded ones.
      • Healy W.L.
      • Bedair H.
      Videoconference interviews for an adult reconstruction fellowship: lessons learned.
      ,
      • Lund S.
      • MacArthur T.
      • Keune J.
      • et al.
      Showcasing a general surgery residency program during the COVID-19 pandemic.
      This leaves only the issue of facilitating interactions between applicants and residents. Some of this gap can be filled by program culture and resident lifestyle content on a website or social media account, which has been shown to have a positive influence on applicants’ perceptions of a program.
      • Labiner H.E.
      • Anderson C.E.
      • Maloney Patel N.
      Virtual recruitment in surgical residency programs.
      ,
      • Duque S.
      • Riccelli V.
      • Mulqueen S.
      • Zhang A.Y.
      Global pandemic and PRS residency match: can social media fill the void?.
      ,
      • Lund S.
      • MacArthur T.
      • Keune J.
      • et al.
      Showcasing a general surgery residency program during the COVID-19 pandemic.
      Interactions with residents and faculty in small breakout rooms and virtual meet-and-greet events such as ours also allow applicants to adequately evaluate culture, morale, and fit.
      • Lewit R.
      • Gosain A.
      Virtual interviews may fall short for pediatric surgery fellowships: lessons learned from COVID-19/SARS-CoV-2.
      ,
      • Lund S.
      • MacArthur T.
      • Keune J.
      • et al.
      Showcasing a general surgery residency program during the COVID-19 pandemic.
      ,
      • Rajesh A.
      • Asaad M.
      Alternative strategies for evaluating general surgery residency applicants and an interview limit for match 2021: an impending necessity.
      ,
      • Buonpane C.
      • Young S.
      • Tuliszewski R.
      • et al.
      Do's and don'ts of the virtual interview: perspectives from residency and fellowship applicants.
      As suggested by multiple applicants in our results, programs may consider utilizing two rounds of interviews. After an initial round of virtual interviews, secondary invites can be sent to the applicants that programs remain interested in, with the option of attending these in person, thus significantly lowering the number, and thereby cost of interviews for both parties. It is crucial, however, that programs are conscious and truly consider those who chose the virtual option versus the in-person equally, thus maintaining the even playing field for financially constrained applicants created by the virtual process. These methods may have their own disadvantages, such as having to plan two separate invitation and interview cycles, or applicants feeling pressured that choosing a virtual option signals disinterest. In fact, the Association of American Medical Colleges has recently recommended against hybrid interviews, specifically fearing pressure to attend an optional in-person sessions and recreating the inequality faced by applicants with financial constraints during in-person only interviews.
      Colleges, T.A.o.A.M
      AAMC interview guidance for the 2022-2023 residency cycle.
      It is thus up to programs to ensure applicants are ranked equitably, regardless of their interview format, even if this involves a “second look” after program rank lists are submitted, serving only applicant rank lists. We believe that, while a hybrid format may require more work to implement well, it could benefit both applicants and programs without loss of the ability to evaluate each other adequately.
      Our study has several limitations. Primarily, the survey was conducted at a single institution with a small sample size, and is limited by a response rate of only 31.2%. There was no in-person interview group to which the current responses could be compared. Although responses were anonymous, applicants may have provided favorable responses due to being local, having prior familiarity with the program, or fear of consequences with negative feedback, thus creating a response bias. Applicants with negative opinions or those who decided not to rank this program may have also chosen not to respond. We neither know the results of respondents' matches nor where applicants’ matches were ranked. The single-question surveys of our matched PGY-1 residents and faculty interviewers also had very small sample sizes and may be biased by them working at our institution. Cost analysis did not factor in differences in the time spent with setup, organization, and conduction of virtual interviews. The cost of additions to further virtual interview seasons such as in-person second looks would also increase the costs saved compared to the 2020-2021 season.

      Conclusions

      The COVID-19 pandemic has proved an enormous challenge for both residency applicants and programs, with both forced to quickly adapt to an unfamiliar format. We have yet to see the long-term impact of the changes in medical education caused by the pandemic and are not sure when we will be safely able to conduct in-person interviews again. It is imperative to use lessons learned in the past two cycles to improve the process moving forward. The virtual interview is an effective method for applicants and programs to evaluate each other and may be a useful adjunct to future interview cycles. Although virtual interviews may not completely replace in-person ones, their addition to the process would significantly lower the costs for applicants and residency programs without sacrificing key elements of the interview.

      Author Contributions

      Dennis M. Vaysburg: project development, data acquisition, data analysis, writing, revisions. Aaron M. Delman: data analysis, survey creation and distribution, revisions, Allison M. Ammann: survey creation and distribution, data acquisition. Kevin M. Turner: data analysis, Leah K. Winer: writing, revision Jeffrey J. Sussman: project development, survey creation, revisions. Amy T. Makley: survey creation, data collection. Michael D. Goodman: survey creation, data collection, revision Ralph C. Quillin: project development, revision Robert M. Van Haren: project.

      Disclosure

      None declared.

      Funding

      None.

      References

        • Pourmand A.
        • Lee H.
        • Fair M.
        • Maloney K.
        • Caggiula A.
        Feasibility and usability of tele-interview for medical residency interview.
        West J Emerg Med. 2018; 19: 80-86
        • Wolff M.
        • Burrows H.
        Planning for virtual interviews: residency recruitment during a pandemic.
        Acad Pediatr. 2021; 21: 24-31
        • Azizzadeh A.
        • McCollum C.H.
        • Miller C.C.
        • Holliday K.M.
        • Shilstone H.C.
        • Lucci A.
        Factors influencing career choice among medical students interested in surgery.
        Curr Surg. 2003; 60: 210-213
        • Kim R.H.
        • Gilbert T.
        • Suh S.
        • Miller J.K.
        • Eggerstedt J.M.
        General surgery residency interviews: are we following best practices?.
        Am J Surg. 2016; 211: 476-481.e3
        • Mayer K.L.
        • Perez R.V.
        • Ho H.S.
        Factors affecting choice of surgical residency training program.
        J Surg Res. 2001; 98: 71-75
        • Sinno S.
        • Mehta K.
        • Squitieri L.
        • et al.
        Residency characteristics that matter most to plastic surgery applicants: a multi-institutional analysis and review of the literature.
        Ann Plast Surg. 2015; 74: 713-717
        • Zuo K.J.
        • Retrouvey H.
        • Wanzel K.R.
        Factors that affect medical students' perception and impression of a plastic surgery program: the role of elective rotations and interviews.
        Ann Plast Surg. 2019; 82: 224-228
        • Labiner H.E.
        • Anderson C.E.
        • Maloney Patel N.
        Virtual recruitment in surgical residency programs.
        Curr Surg Rep. 2021; 9: 25
        • Ammann A.M.
        • Cortez A.R.
        • Vaysburg D.M.
        • et al.
        Examining the impact of COVID-19 restrictions on the operative volumes of US general surgery residents.
        Surgery. 2021; 171: 354-359
        • Van Haren R.M.
        • Delman A.M.
        • Turner K.M.
        • et al.
        Impact of the COVID-19 pandemic on lung cancer screening program and subsequent lung cancer.
        J Am Coll Surg. 2021; 232: 600-605
        • Zaki M.M.
        • Nahed B.V.
        Utilizing virtual interviews in residency selection beyond COVID-19.
        Acad Med. 2020; 95: e7-e8
        • Duque S.
        • Riccelli V.
        • Mulqueen S.
        • Zhang A.Y.
        Global pandemic and PRS residency match: can social media fill the void?.
        Aesthet Surg J. 2021; 41: NP1747-NP1753
        • Majumder A.
        • Eckhouse S.R.
        • Brunt L.M.
        • et al.
        Initial experience with a virtual platform for advanced gastrointestinal minimally invasive surgery fellowship interviews.
        J Am Coll Surg. 2020; 231: 670-678
        • Nguyen J.K.
        • Shah N.
        • Heitkamp D.E.
        • Gupta Y.
        COVID-19 and the radiology match: a residency program's survival guide to the virtual interview season.
        Acad Radiol. 2020; 27: 1294-1297
        • Taylor M.
        • Wallen T.
        • Mehaffey J.H.
        • et al.
        Interviews during the pandemic: a thoracic education cooperative group and surgery residents project.
        Ann Thorac Surg. 2021; 113: 663-668
        • Vining C.C.
        • Eng O.S.
        • Hogg M.E.
        • et al.
        Virtual surgical fellowship recruitment during COVID-19 and its implications for resident/fellow recruitment in the future.
        Ann Surg Oncol. 2020; 27: 911-915
        • D'Angelo J.D.
        • D'Angelo A.L.D.
        • Mathis K.L.
        • Dozois E.J.
        • Kelley S.R.
        Program director opinions of virtual interviews: whatever makes my partners happy.
        J Surg Educ. 2021; 78: e12-e18
        • Daram S.R.
        • Wu R.
        • Tang S.J.
        Interview from anywhere: feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process.
        Am J Gastroenterol. 2014; 109: 155-159
        • Edje L.
        • Miller C.
        • Kiefer J.
        • Oram D.
        Using skype as an alternative for residency selection interviews.
        J Grad Med Educ. 2013; 5: 503-505
        • Grova M.M.
        • Donohue S.J.
        • Meyers M.O.
        • Kim H.J.
        • Ollila D.W.
        Direct comparison of in-person versus virtual interviews for complex general surgical oncology fellowship in the COVID-19 era.
        Ann Surg Oncol. 2021; 28: 1908-1915
        • Healy W.L.
        • Bedair H.
        Videoconference interviews for an adult reconstruction fellowship: lessons learned.
        J Bone Joint Surg Am. 2017; 99: e114
        • Lewit R.
        • Gosain A.
        Virtual interviews may fall short for pediatric surgery fellowships: lessons learned from COVID-19/SARS-CoV-2.
        J Surg Res. 2021; 259: 326-331
        • Lund S.
        • MacArthur T.
        • Keune J.
        • et al.
        Showcasing a general surgery residency program during the COVID-19 pandemic.
        J Surg Educ. 2021; 78: e28-e34
        • Phillips B.T.
        • Gosman A.A.
        • Maricevich R.S.
        • Rodriguez E.D.
        • Rohrich R.J.
        The plastic surgery residency interview revisited: virtual interviews and beyond.
        Plast Reconstr Surg. 2020; 146: 1209-1211
        • Rajesh A.
        • Asaad M.
        • Elmorsi R.
        • Ferry A.M.
        • Maricevich R.S.
        The virtual interview experience for match 2021: a pilot survey of general surgery residency program directors.
        Am Surg. 2021; (31348211038555)
        • Leape L.L.
        • Brennan T.A.
        • Laird N.
        • et al.
        The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II.
        N Engl J Med. 1991; 324: 377-384
        • Walsh K.
        Why is medical education so expensive?.
        J Biomed Res. 2014; 28: 326-327
        • American Association of Medical Colleges, W.D
        AAMC diversity in medical education facts and figures: current trends in medical education.
        (Available at:)
        • Pletcher S.D.
        • Chang C.W.D.
        • Thorne M.C.
        • Malekzadeh S.
        The otolaryngology residency program preference signaling experience.
        Acad Med. 2021; 97: 664-668
        • Salehi P.P.
        • Azizzadeh B.
        • Lee Y.H.
        Preference signaling for competitive residency programs in the NRMP.
        J Grad Med Educ. 2019; 11: 733-734
        • Morgan H.K.
        • Winkel A.F.
        • Standiford T.
        • et al.
        The case for capping residency interviews.
        J Surg Educ. 2021; 78: 755-762
        • Fogel H.A.
        • Liskutin T.E.
        • Wu K.
        • Nystrom L.
        • Martin B.
        • Schiff A.
        The economic burden of residency interviews on applicants.
        Iowa Orthop J. 2018; 38: 9-15
        • Rajesh A.
        • Asaad M.
        Alternative strategies for evaluating general surgery residency applicants and an interview limit for match 2021: an impending necessity.
        Ann Surg. 2021; 273: 109-111
        • Buonpane C.
        • Young S.
        • Tuliszewski R.
        • et al.
        Do's and don'ts of the virtual interview: perspectives from residency and fellowship applicants.
        J Grad Med Educ. 2020; 12: 671-673
        • Colleges, T.A.o.A.M
        AAMC interview guidance for the 2022-2023 residency cycle.
        (Available at:)