THE UTILITY OF A PERSONALISED RISK CALCULATOR IN GYNAE-ONCOLOGY SURGERY

Objective: The objective of this study was to assess the ability of the American College of Surgeons (ACS)
NSQIP surgical risk calculator to accurately identify patients at increased risk of perioperative complication
following surgery for gynaecological malignancy.
Methods: A retrospective review of 142 patients who underwent major surgery under the gynae-oncology
team between 06/08/2018-16/04/2019 at the University Hospital of Wales. Pre-operative factors combined
with a procedure-specific code generated the predicted risk of 13 post-operative complications for each
patient. Brier scores assessed calibration and receiver operated curves (AUC) evaluated the discriminative
power of NSQIP.
Results: Complications were experienced by 50/142 (35.2%) patients. The calculator displayed adequate
calibration when used to predict serious complications (Brier = 0.070), readmission (Brier = 0.058), return
to OR (Brier = 0.000) and UTI (Brier = 0.001). It had the greatest discriminative power when predicting the
risk of serious complications (AUC = 0.672; 95% CI, 0.481-0.863). The calculator successfully identified a
majority of patients who had a complication as being of ‘above average risk’ for all complications, apart
from return to OR, based on their pre-operative factors.
Discussion: NSQIP has previously been demonstrated to be a useful pre-operative tool for evaluating the
risk of post-operative complications in colorectal surgery. This study suggests that in the setting of gynaeoncology surgery the calculator does not have adequate discriminative power to be an absolute predictor of
all complications, however, it may be useful in identifying patients who are likely to develop serious
complications and those at above average risk of complications.


Introduction
Equipping clinicians and patients with accurate information is paramount to making informed decisions regarding surgery. In gynaecology oncology, the patient population has become increasingly challenging to operate on due to the advancing age of the surgical cohort as well as the rising prevalence of obesity and other co-morbidities. There has been a strong link established between the occurrence of postoperative complications and decreased long term survival, thus it is more important than ever to be able to counsel patients appropriately regarding their surgical risk [1].
In the UK the recent launch of the Perioperative Quality Improvement Programme (PQIP) has created a national drive for the use of personalised risk evaluation to improve patient care and outcomes [2]. Cardiopulmonary exercise testing (CPET) is a popular, objective method of achieving this, however the CPET is expensive and oversubscribed [3]. Several online risk prediction tools exist that can improve accuracy of risk assessment in a patient without the need to access CPET. These tools aim to consider patient specific variables in the context of large databases of information in order to generate a form of risk assessment. Tools, such as P-POSSOM and the Surgical Outcome Risk Tool (SORT), are two of the more commonly used tools in general surgery however, they are specifically based on the general surgical population and somewhat limited by the fact that the risk assessment is not operation specific [4][5][6]. Additionally, P-POSSUM has been shown to overestimate risk in patients who had a predicted mortality of ~15% by a factor of two in the NELA trial, suggesting that the tool may be outdated [7]. Risk in gynaecological oncology surgery differs to that in the general surgical population [8]. By definition these patients all have cancer and therefore factors such as poor nutritional status and the physiological impact of the malignancy itself play a significant role [9]. Personalised risk assessment is clearly desirable and is increasingly considered a standard of care, therefore, identifying an accessible tool that is tailored to gynaecology oncology is important.
The American College of Surgeons (ACS) have used data from over 4.3 million operations from the National Surgical Quality Improvement Program (NSQIP) to generate a free web-based personalised surgical risk calculator to address this need. The calculator was originally validated in colorectal surgery and has since been expanded across all surgical sub-specialties to generate the first universal risk calculator [10,11]. NSQIP uses 21 patient predictors, combined with a current procedural terminology (CPT) code to calculate the personalised risk (%) of the patient having any of 13 complications in the 30-day postoperative period. The patient's individualised risk score is then compared to an 'average patient' and stratified as above average, average or below average risk.
Previous research analysing the use of NSQIP in gynaecology oncology is limited and heterogeneous in findings. Of note, Rivard C et al. [7]. looked at the accuracy of the NSQIP calculator specifically in laparotomies performed in gynaecology oncology and found that higher complication scores were associated with an increased risk of actual complication occurring for all events. Teoh D et al. specifically looked at the accuracy of the calculator in minimally invasive gynaecology oncology and found that the calculator did not accurately predict complications [12]. These studies each focused on separate procedures, were performed in non-UK settings, and were conducted at a time when the number of gynaecology oncology patients informing the database was relatively fewer compared to the present day. This study represents and up to date evaluation of the accuracy of the calculator in a UK population specific of all major gynaecology oncology surgeries.

Materials and Methods
A retrospective review of patients who underwent gynaecology oncology surgery at the University Hospital Wales between 06/08/2018-16/04/2019 was performed. Operations of interest were; total laparoscopic hysterectomy with bilateral salpingo-oophorectomy (TLH BSO), total abdominal hysterectomy with bilateral salpingooophorectomy (TAH BSO), bilateral or unilateral salpingooophorectomy (B/USO), interval debulking surgery and vulvectomy. These cases were identified via hospital records, assigned an appropriate current procedural terminology (CPT) code and entered into the ACS NSQIP online calculator http://riskcalculator.facs.org/RiskCalculator/. The predicted risks were noted and compared to the post-operative complications recorded in clinical notes. NSQIP definitions were adhered to for all other analysis.
Statistical analyses were carried out in SPSS Statistics v25 and Excel v15.29. The Brier Score was used to evaluate the difference between predicted risk and recorded complications, and provides a good reference for calibration, i.e. as to whether the model under or overestimates complications. The score ranges between 0, suggesting the predictions are completely accurate, to 1 if they are totally inaccurate. A Brier Score of ≤0.09 was considered as a threshold for sufficient accuracy [12]. How well the calculator discriminated between patients who did/did not have a complication was analysed using the 'Area under the receiver operating curve (AUC)'. A perfect model would have a value of 1, while a value of 0.5 suggests that the predictive power of the calculator is no greater than chance. A model with an AUC >0.7 is generally considered acceptable [13].
The Shapiro-Wilk test revealed that non-parametric tests were most appropriate for the data set. Relative risk and Fisher's exact test were used where appropriate to assess the ability of the calculator to risk stratify patients. A Pearson's rank was used for analysis of predicted length of stay, with one extreme outlier excluded from the calculation. p-values <0.05 were considered statistically significant.

I Demographics
A total of 142 surgeries were carried out over 8 months. One patient was operated on twice in this time and was counted as two separate entries. The average age of the cohort was 60.7 years and average BMI was 30.6. The majority of patients had mild systemic disease (ASA 2). In the cohort there were no patients recorded who were; ventilator dependent, had acute renal failure, congestive heart failure or were on dialysis preoperatively.

II Assessment of the Calculator's Ability to Predict Post-Operative Complications
For all outcomes aforementioned (aside return to operating theatre), the majority of patients experiencing the complication were identified as being above average risk by the calculator. When analysing all patients who had 'any complication', NSQIP allocated more than half of patients to be 'above average' risk (25/46, 54.35%). This was also true when analysing patients who had serious complications (6/10, 60%), SSIs (11/22, 50%), readmission (4/7, 57.4%) and UTI (5/7, 71.43%) ( Figure  1).
Overall, NSQIP was well calibrated with complications in gynaecology oncology surgeries as all Brier scores were <0.4. The difference was most marked in analyses of serious complications (difference in median predicted risk = 5.6%), however this did not reach statistical significance (U = 786.00, p = 0.071) due to considerable overlap of ranges. The calculator appeared to be most accurate at predicting return to OR (Brier = 0.000) and UTI (Brier = 0.001). The calculator was weakest at predicting any complication overall (Brier = 0.31) ( Table 2).
NSQIP had the greatest discriminative power when predicting the risk of serious complications (AUC = 0.672; 95% CI, 0.481-0.863). The calculator had moderate discriminative power in predicting readmission and any complication, however performed poorly in predicting SSI and return to OR. The prediction of risk for UTI was particularly weak (AUC = 0.461; 95% CI, 0.245-0.676) (Figure 2).

III Subgroup Analysis of Patients who Underwent TLH BSO or TAH BSO
The ability of the calculator to risk stratify patients who underwent the two most commonly performed operations was analysed. The TLH BSO and TAH BSO subgroups were stratified into below average risk, average risk and above average risk per the NSQIP predictions for any complication and serious complications. Of the patients who underwent TLH BSO and had any complication, 55% (11/20) were predicted to be at above average risk of developing complications. Only 4 patients in the TLH BSO group developed a serious complication (4/59, 7.41%), and 100% of these patients were predicted to be at above average risk based on pre-operative risk factors. In the TAH BSO cohort 64% (9/14) patients who experienced any complication were predicted to be above average risk whilst, of the 3 patients who underwent this procedure and developed serious complications, 66.7% (2/3) were identified by NSQIP as being of above average risk.

IV Assessment of The Calculator's Ability to Accurately Predict Length of Hospital Stay
Regression analysis was carried out on the whole cohort to analyse how well the calculator predicts length of stay. There was a moderately positive correlation between NSQIP predicted length of stay and actual length of stay, r=0.318, p<0.0005, after exclusion of one significant outlier ( Figure 3).

Figure 3:
There was a positive correlation between NSQIP predicted and actual length of stay (r=0.318, p<0.0005).

Discussion
Accurate risk evaluation in the pre-operative setting provides important information for shared decision making as well as the opportunity to identify patients who may benefit from optimisation prior to surgery. Research has shown that patient understanding of the risks accompanying surgery is poor [14]. However, the use of visual decisionmaking aids is associated with greater patient knowledge and satisfaction with their decision to have surgery [15]. NSQIP is a free and easy to use tool which has the potential to address this need for a wide range of operations [16]. There is currently a paucity of data regarding its validity in gynae-oncology, highlighting the importance of analysing NSQIP further in this field.
In this study a retrospective analysis of women who underwent surgery for gynaecological malignancies was carried out to assess how useful the NSQIP calculator is in predicting the risk of complications in the postoperative period and how well the calculator identifies patients who are at above average risk of complications. Overall, post-operative complication rates in this centre were comparable to those of gynaeoncology surgery across the UK [17].
NSQIP was shown to be a useful predictor of serious complications in the first 30 days after surgery. This is clinically significant given that serious complications include life-changing events such as cardiac arrest, renal failure and systemic sepsis all of which are associated with substantial mortality and prolonged hospital admissions [18][19][20]. Further to this, the calculator correctly identified the majority of patients who developed complications as 'above average risk' based on their clinical factors, demonstrating that NSQIP can be used to identify patients who may benefit from pre-operative interventions or closer monitoring. This has been implemented in a US medical centre as part of a multi-system approach, successfully decreasing 30-day mortality by >60% [21]. Both the ability to predict the occurrence of serious complications and assess the patient's overall risk level when compared to an average patient aids decision making when medical treatment is an option as well as surgery.
The risk stratification aspect of the calculator performed particularly well when TAH BSO and TLH BSO were analysed in isolation. As expected, SSI was the most commonly encountered complication for both operations [22]. Though NSQIP is moderately accurate at predicting this complication it fails to account for the difference in clinical implications of an SSI for a TAH vs TLH patient. A large study found that patients undergoing TAH were more likely to have a deep SSI, which is associated high rates of readmission, return to operating theatre and longer lengths of stay, compared to SSI in TLH BSO patients [22,23]. This demonstrates the importance of using NSQIP results alongside clinical context when counselling patients.
In this study NSQIP predicted length of stay correlated positively with actual length of stay. Though the correlation was weak, this is to be expected given that the calculator is based on the US healthcare system which has different discharge patterns to the NHS. This function of the calculator will be useful from a resource allocation perspective as well as giving the patient a more accurate estimation of time in hospital. As seen in previous research the calculator generally overestimated risk in patients who did not develop complications, whilst predicting an appropriate amount of risk for those who did [8]. Though the calculator did not have the discriminatory power to reliably predict most of the complications, this heterogeneity is consistent with existing studies examining NSQIP in gynae-oncology [8,24,25]. This could be due to differences in operations included, as some studies only focussed on specific procedures or malignancies, whilst this study analysed all patients operated on in the time period. The variability could be further explained by the fact that gynaecology procedures only comprise of 5.3% of the NSQIP cohort used to generate the calculator, compared to the 59.4% of cases that general surgical population account for, demonstrating the need for the calculator to be studied further in this field [11]. This study has contributed to an under researched area and aims to raise awareness of the utility of NSQIP and personalised risk evaluation.
There are several limitations to the use of NSQIP in gynae-oncology. Firstly, hysterectomies are split per uterine weight (<250g or >250g), however, this is not a routine radiological measurement used in the UK. Furthermore, specific operation codes were not available for every surgery, therefore substitutions had to be made for a number of cases. In this study, a small sample size meant that there were a number of patients for whom there was insufficient data to generate an NSQIP calculation and extensive subgroup analysis could not be carried out. Further to this, the Brier score was skewed in favour of patients who did not have complications, particularly when analysing UTI, due the comparatively small number of patients who developed complications in the postoperative period. Finally, during data collection there was difficulty in distinguishing patients undergoing interval debulking surgery (IDS) within the TAH BSO cohort. This is significant as patients undergoing IDS have usually undergone multiple rounds of chemotherapy thus have a higher load of co-morbidities compared to the general gynae-oncology population [26,27].
There are some risk models for specific complications in gynaeoncology, such as the need for transfusion after hysterectomy for ovarian cancer, however, there is currently no risk prediction tool which is applicable to a range of operations in this field, demonstrating a major advantage NSQIP [28]. The model is continually being developed and updated thus over time the number of gynaecology cases included will hopefully grow producing more reliable results. If NSQIP is integrated as part of standard practice it would be interesting to see if complication rates change significantly.
In conclusion NSQIP has been shown to be useful in predicting the risk of serious complications and in identifying patients who are at above average risk of complications, however it should be used as part of a wider clinical discussion to guide patient care. The NSQIP is an easily accessible free resource which may be of benefit in evaluating surgical risk in complex patients undergoing gynae-oncology surgery.