The Utility of a Personalised Risk Calculator in Gynae-Oncology Surgery

A B S T R A C T

Objective: The objective of this study was to assess the ability of the American College of Surgeons (ACS) NSQIP surgical risk calculator to accurately identify patients at increased risk of perioperative complication following surgery for gynaecological malignancy.
Methods: A retrospective review of 142 patients who underwent major surgery under the gynae-oncology team between 06/08/2018-16/04/2019 at the University Hospital of Wales. Pre-operative factors combined with a procedure-specific code generated the predicted risk of 13 post-operative complications for each patient. Brier scores assessed calibration and receiver operated curves (AUC) evaluated the discriminative power of NSQIP.
Results: Complications were experienced by 50/142 (35.2%) patients. The calculator displayed adequate calibration when used to predict serious complications (Brier = 0.070), readmission (Brier = 0.058), return to OR (Brier = 0.000) and UTI (Brier = 0.001). It had the greatest discriminative power when predicting the risk of serious complications (AUC = 0.672; 95% CI, 0.481-0.863). The calculator successfully identified a majority of patients who had a complication as being of ‘above average risk’ for all complications, apart from return to OR, based on their pre-operative factors.
Discussion: NSQIP has previously been demonstrated to be a useful pre-operative tool for evaluating the risk of post-operative complications in colorectal surgery. This study suggests that in the setting of gynae-oncology surgery the calculator does not have adequate discriminative power to be an absolute predictor of all complications, however, it may be useful in identifying patients who are likely to develop serious complications and those at above average risk of complications.

Keywords

Surgical risk, gynaecology-oncology

Introduction

Equipping clinicians and patients with accurate information is paramount to making informed decisions regarding surgery. In gynaecology oncology, the patient population has become increasingly challenging to operate on due to the advancing age of the surgical cohort as well as the rising prevalence of obesity and other co-morbidities. There has been a strong link established between the occurrence of post-operative complications and decreased long term survival, thus it is more important than ever to be able to counsel patients appropriately regarding their surgical risk [1].

In the UK the recent launch of the Perioperative Quality Improvement Programme (PQIP) has created a national drive for the use of personalised risk evaluation to improve patient care and outcomes [2]. Cardiopulmonary exercise testing (CPET) is a popular, objective method of achieving this, however the CPET is expensive and oversubscribed [3]. Several online risk prediction tools exist that can improve accuracy of risk assessment in a patient without the need to access CPET. These tools aim to consider patient specific variables in the context of large databases of information in order to generate a form of risk assessment. Tools, such as P-POSSOM and the Surgical Outcome Risk Tool (SORT), are two of the more commonly used tools in general surgery however, they are specifically based on the general surgical population and somewhat limited by the fact that the risk assessment is not operation specific [4-6]. Additionally, P-POSSUM has been shown to overestimate risk in patients who had a predicted mortality of ~15% by a factor of two in the NELA trial, suggesting that the tool may be outdated [7]. Risk in gynaecological oncology surgery differs to that in the general surgical population [8]. By definition these patients all have cancer and therefore factors such as poor nutritional status and the physiological impact of the malignancy itself play a significant role [9]. Personalised risk assessment is clearly desirable and is increasingly considered a standard of care, therefore, identifying an accessible tool that is tailored to gynaecology oncology is important.

The American College of Surgeons (ACS) have used data from over 4.3 million operations from the National Surgical Quality Improvement Program (NSQIP) to generate a free web-based personalised surgical risk calculator to address this need. The calculator was originally validated in colorectal surgery and has since been expanded across all surgical sub-specialties to generate the first universal risk calculator [10, 11]. NSQIP uses 21 patient predictors, combined with a current procedural terminology (CPT) code to calculate the personalised risk (%) of the patient having any of 13 complications in the 30-day post-operative period. The patient’s individualised risk score is then compared to an ‘average patient’ and stratified as above average, average or below average risk.

Previous research analysing the use of NSQIP in gynaecology oncology is limited and heterogeneous in findings. Of note, Rivard C et al. [7]. looked at the accuracy of the NSQIP calculator specifically in laparotomies performed in gynaecology oncology and found that higher complication scores were associated with an increased risk of actual complication occurring for all events. Teoh D et al. specifically looked at the accuracy of the calculator in minimally invasive gynaecology oncology and found that the calculator did not accurately predict complications [12]. These studies each focused on separate procedures, were performed in non-UK settings, and were conducted at a time when the number of gynaecology oncology patients informing the database was relatively fewer compared to the present day. This study represents and up to date evaluation of the accuracy of the calculator in a UK population specific of all major gynaecology oncology surgeries.

Materials and Methods

A retrospective review of patients who underwent gynaecology oncology surgery at the University Hospital Wales between 06/08/2018-16/04/2019 was performed. Operations of interest were; total laparoscopic hysterectomy with bilateral salpingo-oophorectomy (TLH BSO), total abdominal hysterectomy with bilateral salpingo-oophorectomy (TAH BSO), bilateral or unilateral salpingo-oophorectomy (B/USO), interval debulking surgery and vulvectomy. These cases were identified via hospital records, assigned an appropriate current procedural terminology (CPT) code and entered into the ACS NSQIP online calculator http://riskcalculator.facs.org/RiskCalculator/. The predicted risks were noted and compared to the post-operative complications recorded in clinical notes. NSQIP definitions were adhered to for all other analysis.

Statistical analyses were carried out in SPSS Statistics v25 and Excel v15.29. The Brier Score was used to evaluate the difference between predicted risk and recorded complications, and provides a good reference for calibration, i.e. as to whether the model under or overestimates complications. The score ranges between 0, suggesting the predictions are completely accurate, to 1 if they are totally inaccurate. A Brier Score of ≤0.09 was considered as a threshold for sufficient accuracy [12]. How well the calculator discriminated between patients who did/did not have a complication was analysed using the ‘Area under the receiver operating curve (AUC)’. A perfect model would have a value of 1, while a value of 0.5 suggests that the predictive power of the calculator is no greater than chance. A model with an AUC >0.7 is generally considered acceptable [13].

The Shapiro-Wilk test revealed that non-parametric tests were most appropriate for the data set. Relative risk and Fisher’s exact test were used where appropriate to assess the ability of the calculator to risk stratify patients. A Pearson’s rank was used for analysis of predicted length of stay, with one extreme outlier excluded from the calculation. p-values <0.05 were considered statistically significant.

Results

I Demographics

A total of 142 surgeries were carried out over 8 months. One patient was operated on twice in this time and was counted as two separate entries. The average age of the cohort was 60.7 years and average BMI was 30.6. The majority of patients had mild systemic disease (ASA 2). In the cohort there were no patients recorded who were; ventilator dependent, had acute renal failure, congestive heart failure or were on dialysis pre-operatively.

The most commonly performed operation was TLH BSO (59/142, 41.5%), followed by TAH BSO (47/142, 33.1%). Vulval procedures (which included partial and total vulvectomy), U/BSO and interval debulking accounted for 7% (10 /142), 6.3% (9/142) and 3.5% (5/142) of surgeries respectively. The remaining 8.5% (12/142) of the cohort underwent ‘other’ procedures such as trachelectomy and hernia repair. The demographics of the cohort are summarised in (Table 1).

Overall, 35.2% (50/142) of patients experienced one or more complication, and 7% (10/142) experienced a serious complication. The most common complications were surgical site infection (SSI) (24/142, 16.9%), readmission (8/142, 8%), return to operating room (OR) (7/142, 4.9%) and urinary tract infection (UTI) (7/142, 4.9%). Several patients (15/142, 10.6%) experienced ‘other’ complications not specified by the calculator, of which the majority were wound dehiscence. There were no deaths, cardiac complications or pneumonias recorded in the cohort.

Vulval procedures had the highest overall complication rate (5/10, 50%), all of which were accounted for by SSI and wound dehiscence. TLH BSO had marginally greater complication rates (Any; 21/59, 35.6%, Serious; 4/59, 7.41%), compared with TAH BSO (Any; 16/47, 34.0%, Serious; 3/47, 6.4%), however, within both subgroups SSI was the most commonly encountered complication. Bilateral / unilateral Salpingo-oophorectomy had the lowest complication rate (Any; 3/9, 33.3%, Serious; 0/9, 0%).

Table 1: Demographics of patient population, per NSQIP required risk factors (N=142).

	TLH BSO		TAH BSO		Vulval		B/USO		Interval Debulk		Other
	N	%	N	%	N	%	N	%	N	%	N	%
	59	42	47	33	10	7	9	6	5	4	12	8
Age
Under 65 years	31	53	29	62	3	30	6	67	2	40	9	75
65-74 years	16	27	9	19	4	40	2	22	3	60	2	17
75-84 years	10	17	8	17	3	30	1	11	0	0	1	8
85 years or older	2	3	1	2	0	0	0	0	0	0	0	0
Average Age	61.3		62.0		68.8		54.1		63.8		49.3
Functional Status
Independent	56	95	46	98	8	80	8	89	4	80	12	100
Partially dependent	2	3	1	2	1	10	1	11	1	20	0	0
Dependent	1	2	0	0	1	10	0	0	0	0	0	0
Emergency Case
No	59	100	46	98	10	100	9	100	5	100	12	100
Yes	0	0	1	2	0	0	0	0	0	0	0	0
ASA Class
Healthy patient	8	14	4	9	0	0	2	22	0	0	4	33
Mild systemic disease	34	58	23	49	6	60	6	67	2	40	6	50
Severe systemic disease	17	29	20	43	4	40	1	11	3	60	2	17
Constant threat to life/Moribund	0	0	0	0	0	0	0	0	0	0	0	0
Steroid use for chronic condition
No	53	90	38	81	7	70	8	89	4	80	12	100
Yes	6	10	9	19	3	30	1	11	1	20	0	0
Ascites within 30 days prior to surgery
No	58	98	32	68	10	100	6	67	5	100	12	100
Yes	1	2	15	32	0	0	3	33	0	0	0	0
Systemic sepsis
None	59	100	46	98	10	100	9	100	5	100	12	100
Sepsis	0	0	1	2	0	0	0	0	0	0	0	0
Disseminated cancer
No	51	86	27	57	7	70	7	78	1	20	8	67
Yes	8	14	20	43	3	30	2	22	4	80	4	33
Diabetes
No	48	81	45	96	9	90	9	100	5	100	11	92
Oral	11	19	2	4	0	0	0	0	0	0	1	8
Insulin	0	0	0	0	1	10	0	0	0	0	0	0
Hypertension requiring medication
No	35	59	30	64	5	50	6	67	3	60	12	100
Yes	24	41	17	36	5	50	3	33	2	40	0	0
Dyspnoea
No	57	97	44	94	10	100	9	100	5	100	12	100
With moderate exertion	2	3	3	6	0	0	0	0	0	0	0	0
At rest	0	0	0	0	0	0	0	0	0	0	0	0
Current smoker within 1 year
No	55	93	41	87	8	80	9	100	5	100	10	83
Yes	4	7	6	13	2	20	0	0	0	0	2	17
History of severe COPD
No	56	95	42	89	10	100	9	100	5	100	12	100
Yes	3	5	5	11	0	0	0	0	0	0	0	0
BMI (kg/m²)
Underweight: <18.5	0	0	0	0	1	10	0	0	0	0	1	8
Healthy: 18.5-24.9	10	17	12	26	1	10	1	11	3	60	3	25
Overweight: 25-29.9	13	22	9	19	2	20	1	11	1	20	3	25
Obesity I: 30-34.9	14	24	11	23	1	10	4	44	0	0	3	25
Obesity II: 35-39.9	6	10	4	9	1	10	1	11	0	0	0	0
Obesity III: ≥40	10	17	3	6	1	10	1	11	0	0	1	8
Unknown	6	10	8	17	3	30	1	11	1	20	1	8
Average BMI	32.4		29.6		30.1		32.1		23.41		27.5
Average Length of Stay (Days)	4.56		5.07		4.1		3.67		7.4		3.83

Table 2: Statistical analysis of NSQIP predicted risk vs. observed complication rates for any complication, serious complications, SSI, readmission, return to OR and UTI.

	Brier Score	Did not have the complication			Had the complication			U Value
		N	%	Median Risk % (Min-Max)	N	%	Median Risk % (Min-Max)
Any complication	0.310	92	64.8	7.2 (1.9-24.0)	50	35.2	8.0 (2.1-29.2)	2017	p=0.465
Serious complication	0.070	132	93.0	6.0 (1.5-24.1)	10	7	12.4 (3.2-18.0)	786	p=0.071
SSI	0.168	118	83.1	3 (1-15)	24	16.9	3.5 (1.0-9.4)	1266.5	p=0.659
Readmission	0.058	134	94.4	4.1 (0.8-16.8)	8	5.6	7.2 (2.0-12.6)	620	p=0.153
Return to OR	0.000	135	95.1	1.3 (0.3-5.4)	7	4.9	1.7 (0.8-2.5)	461.5	p=0.661
UTI	0.001	135	95.1	2.9 (0.3-10.3)	7	4.9	2.9 (1.4-6.3)	387	p=0.727

II Assessment of the Calculator’s Ability to Predict Post-Operative Complications

For all outcomes aforementioned (aside return to operating theatre), the majority of patients experiencing the complication were identified as being above average risk by the calculator. When analysing all patients who had ‘any complication’, NSQIP allocated more than half of patients to be ‘above average’ risk (25/46, 54.35%). This was also true when analysing patients who had serious complications (6/10, 60%), SSIs (11/22, 50%), readmission (4/7, 57.4%) and UTI (5/7, 71.43%) (Figure 1).

Overall, NSQIP was well calibrated with complications in gynaecology oncology surgeries as all Brier scores were <0.4. The difference was most marked in analyses of serious complications (difference in median predicted risk = 5.6%), however this did not reach statistical significance (U = 786.00, p = 0.071) due to considerable overlap of ranges. The calculator appeared to be most accurate at predicting return to OR (Brier = 0.000) and UTI (Brier = 0.001). The calculator was weakest at predicting any complication overall (Brier = 0.31) (Table 2).

NSQIP had the greatest discriminative power when predicting the risk of serious complications (AUC = 0.672; 95% CI, 0.481-0.863). The calculator had moderate discriminative power in predicting readmission and any complication, however performed poorly in predicting SSI and return to OR. The prediction of risk for UTI was particularly weak (AUC = 0.461; 95% CI, 0.245-0.676) (Figure 2).

III Subgroup Analysis of Patients who Underwent TLH BSO or TAH BSO

The ability of the calculator to risk stratify patients who underwent the two most commonly performed operations was analysed. The TLH BSO and TAH BSO subgroups were stratified into below average risk, average risk and above average risk per the NSQIP predictions for any complication and serious complications. Of the patients who underwent TLH BSO and had any complication, 55% (11/20) were predicted to be at above average risk of developing complications. Only 4 patients in the TLH BSO group developed a serious complication (4/59, 7.41%), and 100% of these patients were predicted to be at above average risk based on pre-operative risk factors. In the TAH BSO cohort 64% (9/14) patients who experienced any complication were predicted to be above average risk whilst, of the 3 patients who underwent this procedure and developed serious complications, 66.7% (2/3) were identified by NSQIP as being of above average risk.

IV Assessment of The Calculator’s Ability to Accurately Predict Length of Hospital Stay

Regression analysis was carried out on the whole cohort to analyse how well the calculator predicts length of stay. There was a moderately positive correlation between NSQIP predicted length of stay and actual length of stay, r=0.318, p<0.0005, after exclusion of one significant outlier (Figure 3).

Discussion

Accurate risk evaluation in the pre-operative setting provides important information for shared decision making as well as the opportunity to identify patients who may benefit from optimisation prior to surgery. Research has shown that patient understanding of the risks accompanying surgery is poor [14]. However, the use of visual decision-making aids is associated with greater patient knowledge and satisfaction with their decision to have surgery [15]. NSQIP is a free and easy to use tool which has the potential to address this need for a wide range of operations [16]. There is currently a paucity of data regarding its validity in gynae-oncology, highlighting the importance of analysing NSQIP further in this field.

In this study a retrospective analysis of women who underwent surgery for gynaecological malignancies was carried out to assess how useful the NSQIP calculator is in predicting the risk of complications in the post-operative period and how well the calculator identifies patients who are at above average risk of complications. Overall, post-operative complication rates in this centre were comparable to those of gynae-oncology surgery across the UK [17].

NSQIP was shown to be a useful predictor of serious complications in the first 30 days after surgery. This is clinically significant given that serious complications include life-changing events such as cardiac arrest, renal failure and systemic sepsis all of which are associated with substantial mortality and prolonged hospital admissions [18-20]. Further to this, the calculator correctly identified the majority of patients who developed complications as ‘above average risk’ based on their clinical factors, demonstrating that NSQIP can be used to identify patients who may benefit from pre-operative interventions or closer monitoring. This has been implemented in a US medical centre as part of a multi-system approach, successfully decreasing 30-day mortality by >60% [21]. Both the ability to predict the occurrence of serious complications and assess the patient’s overall risk level when compared to an average patient aids decision making when medical treatment is an option as well as surgery.

The risk stratification aspect of the calculator performed particularly well when TAH BSO and TLH BSO were analysed in isolation. As expected, SSI was the most commonly encountered complication for both operations [22]. Though NSQIP is moderately accurate at predicting this complication it fails to account for the difference in clinical implications of an SSI for a TAH vs TLH patient. A large study found that patients undergoing TAH were more likely to have a deep SSI, which is associated high rates of readmission, return to operating theatre and longer lengths of stay, compared to SSI in TLH BSO patients [22, 23]. This demonstrates the importance of using NSQIP results alongside clinical context when counselling patients.

In this study NSQIP predicted length of stay correlated positively with actual length of stay. Though the correlation was weak, this is to be expected given that the calculator is based on the US healthcare system which has different discharge patterns to the NHS. This function of the calculator will be useful from a resource allocation perspective as well as giving the patient a more accurate estimation of time in hospital. As seen in previous research the calculator generally overestimated risk in patients who did not develop complications, whilst predicting an appropriate amount of risk for those who did [8]. Though the calculator did not have the discriminatory power to reliably predict most of the complications, this heterogeneity is consistent with existing studies examining NSQIP in gynae-oncology [8, 24, 25]. This could be due to differences in operations included, as some studies only focussed on specific procedures or malignancies, whilst this study analysed all patients operated on in the time period. The variability could be further explained by the fact that gynaecology procedures only comprise of 5.3% of the NSQIP cohort used to generate the calculator, compared to the 59.4% of cases that general surgical population account for, demonstrating the need for the calculator to be studied further in this field [11]. This study has contributed to an under researched area and aims to raise awareness of the utility of NSQIP and personalised risk evaluation.

There are several limitations to the use of NSQIP in gynae-oncology. Firstly, hysterectomies are split per uterine weight (<250g or >250g), however, this is not a routine radiological measurement used in the UK. Furthermore, specific operation codes were not available for every surgery, therefore substitutions had to be made for a number of cases. In this study, a small sample size meant that there were a number of patients for whom there was insufficient data to generate an NSQIP calculation and extensive subgroup analysis could not be carried out. Further to this, the Brier score was skewed in favour of patients who did not have complications, particularly when analysing UTI, due the comparatively small number of patients who developed complications in the post-operative period. Finally, during data collection there was difficulty in distinguishing patients undergoing interval debulking surgery (IDS) within the TAH BSO cohort. This is significant as patients undergoing IDS have usually undergone multiple rounds of chemotherapy thus have a higher load of co-morbidities compared to the general gynae-oncology population [26, 27].

There are some risk models for specific complications in gynae-oncology, such as the need for transfusion after hysterectomy for ovarian cancer, however, there is currently no risk prediction tool which is applicable to a range of operations in this field, demonstrating a major advantage NSQIP [28]. The model is continually being developed and updated thus over time the number of gynaecology cases included will hopefully grow producing more reliable results. If NSQIP is integrated as part of standard practice it would be interesting to see if complication rates change significantly.

In conclusion NSQIP has been shown to be useful in predicting the risk of serious complications and in identifying patients who are at above average risk of complications, however it should be used as part of a wider clinical discussion to guide patient care. The NSQIP is an easily accessible free resource which may be of benefit in evaluating surgical risk in complex patients undergoing gynae-oncology surgery.

Highlights

• Personalised risk evaluation is becoming increasingly more important in the pre-operative setting.
• NSQIP predicts serious complications well and identifies patients who are at an increased risk of complications.
• NSQIP is free, easy to use and has a role in multidisciplinary discussion when making decisions regarding surgery.

Article Info

Article Type

Research Article

Publication history

Received: Thu 16, Apr 2020
Accepted: Mon 04, May 2020
Published: Fri 27, Nov 2020

Copyright

© 2023 Sadie Jones. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Hosting by Science Repository.
DOI: 10.31487/j.COR.2020.05.04

Author Info

Inés Murray Kenneth Lim Robert Howells Rhidian Jones Aarti Sharma Sadie Jones

Corresponding Author

Sadie Jones
Department of Gynaecology Oncology, University Hospital Wales, Cardiff, Wales, UK

Figures & Tables

Table 1: Demographics of patient population, per NSQIP required risk factors (N=142).

	TLH BSO		TAH BSO		Vulval		B/USO		Interval Debulk		Other
	N	%	N	%	N	%	N	%	N	%	N	%
	59	42	47	33	10	7	9	6	5	4	12	8
Age
Under 65 years	31	53	29	62	3	30	6	67	2	40	9	75
65-74 years	16	27	9	19	4	40	2	22	3	60	2	17
75-84 years	10	17	8	17	3	30	1	11	0	0	1	8
85 years or older	2	3	1	2	0	0	0	0	0	0	0	0
Average Age	61.3		62.0		68.8		54.1		63.8		49.3
Functional Status
Independent	56	95	46	98	8	80	8	89	4	80	12	100
Partially dependent	2	3	1	2	1	10	1	11	1	20	0	0
Dependent	1	2	0	0	1	10	0	0	0	0	0	0
Emergency Case
No	59	100	46	98	10	100	9	100	5	100	12	100
Yes	0	0	1	2	0	0	0	0	0	0	0	0
ASA Class
Healthy patient	8	14	4	9	0	0	2	22	0	0	4	33
Mild systemic disease	34	58	23	49	6	60	6	67	2	40	6	50
Severe systemic disease	17	29	20	43	4	40	1	11	3	60	2	17
Constant threat to life/Moribund	0	0	0	0	0	0	0	0	0	0	0	0
Steroid use for chronic condition
No	53	90	38	81	7	70	8	89	4	80	12	100
Yes	6	10	9	19	3	30	1	11	1	20	0	0
Ascites within 30 days prior to surgery
No	58	98	32	68	10	100	6	67	5	100	12	100
Yes	1	2	15	32	0	0	3	33	0	0	0	0
Systemic sepsis
None	59	100	46	98	10	100	9	100	5	100	12	100
Sepsis	0	0	1	2	0	0	0	0	0	0	0	0
Disseminated cancer
No	51	86	27	57	7	70	7	78	1	20	8	67
Yes	8	14	20	43	3	30	2	22	4	80	4	33
Diabetes
No	48	81	45	96	9	90	9	100	5	100	11	92
Oral	11	19	2	4	0	0	0	0	0	0	1	8
Insulin	0	0	0	0	1	10	0	0	0	0	0	0
Hypertension requiring medication
No	35	59	30	64	5	50	6	67	3	60	12	100
Yes	24	41	17	36	5	50	3	33	2	40	0	0
Dyspnoea
No	57	97	44	94	10	100	9	100	5	100	12	100
With moderate exertion	2	3	3	6	0	0	0	0	0	0	0	0
At rest	0	0	0	0	0	0	0	0	0	0	0	0
Current smoker within 1 year
No	55	93	41	87	8	80	9	100	5	100	10	83
Yes	4	7	6	13	2	20	0	0	0	0	2	17
History of severe COPD
No	56	95	42	89	10	100	9	100	5	100	12	100
Yes	3	5	5	11	0	0	0	0	0	0	0	0
BMI (kg/m²)
Underweight: <18.5	0	0	0	0	1	10	0	0	0	0	1	8
Healthy: 18.5-24.9	10	17	12	26	1	10	1	11	3	60	3	25
Overweight: 25-29.9	13	22	9	19	2	20	1	11	1	20	3	25
Obesity I: 30-34.9	14	24	11	23	1	10	4	44	0	0	3	25
Obesity II: 35-39.9	6	10	4	9	1	10	1	11	0	0	0	0
Obesity III: ≥40	10	17	3	6	1	10	1	11	0	0	1	8
Unknown	6	10	8	17	3	30	1	11	1	20	1	8
Average BMI	32.4		29.6		30.1		32.1		23.41		27.5
Average Length of Stay (Days)	4.56		5.07		4.1		3.67		7.4		3.83

Table 2: Statistical analysis of NSQIP predicted risk vs. observed complication rates for any complication, serious complications, SSI, readmission, return to OR and UTI.

	Brier Score	Did not have the complication			Had the complication			U Value
		N	%	Median Risk % (Min-Max)	N	%	Median Risk % (Min-Max)
Any complication	0.310	92	64.8	7.2 (1.9-24.0)	50	35.2	8.0 (2.1-29.2)	2017	p=0.465
Serious complication	0.070	132	93.0	6.0 (1.5-24.1)	10	7	12.4 (3.2-18.0)	786	p=0.071
SSI	0.168	118	83.1	3 (1-15)	24	16.9	3.5 (1.0-9.4)	1266.5	p=0.659
Readmission	0.058	134	94.4	4.1 (0.8-16.8)	8	5.6	7.2 (2.0-12.6)	620	p=0.153
Return to OR	0.000	135	95.1	1.3 (0.3-5.4)	7	4.9	1.7 (0.8-2.5)	461.5	p=0.661
UTI	0.001	135	95.1	2.9 (0.3-10.3)	7	4.9	2.9 (1.4-6.3)	387	p=0.727

Figure 1: A clustered bar chart demonstrating the predicted risk allocation of patients who experienced complications, based on their pre-operative factors.

Figure 2: The area under the curve (AUC) represents the discriminatory power of a model, where the diagonal line represents chance. A perfect model would run vertically up the y axis from 0 and plateau at 1. a) Any complication (AUC = 0.548; 95% CI 0.443-0.654), b) Serious complication (AUC = 0.672; 95% CI 0.481-0.863), c) SSI (AUC = 0.529; 95% CI 0.391-0.668), d) Readmission (AUC =0.650; 95% CI 0.474-0.825), e) Return to OR (AUC = 0.549; 95% CI 0.339-0.760) ) and f) UTI (AUC = 0.461; 95% CI 0.245-0.676).

Figure 3: There was a positive correlation between NSQIP predicted and actual length of stay (r=0.318, p<0.0005).

References

1. Khuri SF, Henderson WG, DePalma RG, Mosca C, Healey NA et al. (2005) Determinants of long-term survival after major surgery and the adverse effect of postoperative complications. Ann Surg 242: 326-341. [Crossref]
2. Sahni AaMR (2018) The Perioperative Quality Improvement Programme. The Bulletin of the Royal College of Surgeons of England 100: 335-337.
3. Reeves T, Bates S, Sharp T, Richardson K, Bali S et al. (2018) Cardiopulmonary exercise testing (CPET) in the United Kingdom-a national survey of the structure, conduct, interpretation and funding. Perioper Med (Lond) 7: 2. [Crossref]
4. Prytherch DR, Whiteley MS, Higgins B, Weaver PC, Prout WG et al. (1998) POSSUM and Portsmouth POSSUM for predicting mortality. Physiological and Operative Severity Score for the enUmeration of Mortality and morbidity. Br J Surg 85: 1217-1220. [Crossref]
5. Protopapa KL, Simpson JC, Smith NC, Moonesinghe SR (2014) Development and validation of the Surgical Outcome Risk Tool (SORT). Br J Surg 101: 1774-1783. [Crossref]
6. Moonesinghe SR, Mythen MG, Das P, Rowan KM, Grocott MP (2013) Risk stratification tools for predicting morbidity and mortality in adult patients undergoing major surgery: qualitative systematic review. Anesthesiology 119: 959-981. [Crossref]
7. Team NP (2016) The Second Patient Report of the National Emergency Laparotomy Audit (NELA). RCoA London.
8. Rivard C, Nahum R, Slagle E, Duininck M, Isaksson Vogel R et al. (2016) Evaluation of the performance of the ACS NSQIP surgical risk calculator in gynecologic oncology patients undergoing laparotomy. Gynecol Oncol 141: 281-286. [Crossref]
9. Laky B, Janda M, Bauer J, Vavra C, Cleghorn G et al. (2007) Malnutrition among gynaecological cancer patients. Eur J Clin Nutr 61: 642-646. [Crossref]
10. Cohen ME, Bilimoria KY, Ko CY, Hall BL (2009) Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery. J Am Coll Surg 208: 1009-1016. [Crossref]
11. Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE et al. (2013) Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 217: 833-842. [Crossref]
12. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M et al. (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21: 128-138. [Crossref]
13. Mandrekar JN (2010) Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol 5: 1315-1316. [Crossref]
14. Mulsow JJ, Feeley TM, Tierney S (2012) Beyond consent--improving understanding in surgical patients. Am J Surg 203: 112-120. [Crossref]
15. Whelan T, Levine M, Willan A, Gafni A, Sanders K et al. (2004) Effect of a decision aid on knowledge and treatment decision making for breast cancer surgery: a randomized trial. JAMA 292: 435-441. [Crossref]
16. Raymond BL, Wanderer JP, Hawkins AT, Geiger TM, Ehrenfeld JM et al. (2019) Use of the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator During Preoperative Risk Discussion: The Patient Perspective. Anesth Analg 128: 643-650. [Crossref]
17. Burnell M, Iyer R, Gentry Maharaj A, Nordin A, Liston R et al. (2016) Benchmarking of surgical complications in gynaecological oncology: prospective multicentre study. BJOG 123: 2171-2180. [Crossref]
18. Kazaure HS, Roman SA, Rosenthal RA, Sosa JA (2013) Cardiac arrest among surgical patients: an analysis of incidence, patient characteristics, and outcomes in ACS-NSQIP. JAMA Surg 148: 14-21. [Crossref]
19. Gameiro J, Fonseca JA, Neves M, Jorge S, Lopes JA (2018) Acute kidney injury in major abdominal surgery: incidence, risk factors, pathogenesis and outcomes. Ann Intensive Care 8: 22. [Crossref]
20. Ou L, Chen J, Hillman K, Flabouris A, Parr M et al. (2017) The impact of post-operative sepsis on mortality after hospital discharge among elective surgical patients: a population-based cohort study. Crit Care 21: 34. [Crossref]
21. Kuy S, Romero RAL (2017) Decreasing 30-day surgical mortality in a VA Medical Center utilizing the ACS NSQIP Surgical Risk Calculator. J Surg Res 215: 28-33. [Crossref]
22. Uppal S, Penn C, Del Carmen MG, Rauh Hain JA, Reynolds RK et al. (2016) Readmissions after major gynecologic oncology surgery. Gynecol Oncol 141: 287-292. [Crossref]
23. Morgan DM, Swenson CW, Streifel KM, Kamdar NS, Uppal S et al. (2016) Surgical site infection following hysterectomy: adjusted rankings in a regional collaborative. Am J Obstet Gynecol 214: 259e1-259e8. [Crossref]
24. Szender JB, Frederick PJ, Eng KH, Akers SN, Lele SB et al. (2015) Evaluation of the National Surgical Quality Improvement Program Universal Surgical Risk Calculator for a gynecologic oncology service. Int J Gynecol Cancer 25: 512-520. [Crossref]
25. Teoh D, Halloway RN, Heim J, Vogel RI, Rivard C et al. (2017) Evaluation of the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator in Gynecologic Oncology Patients Undergoing Minimally Invasive Surgery. J Minim Invasive Gynecol 24: 48-54. [Crossref]
26. Keyver Paik MD, Zivanovic O, Rudlowski C, Höller T, Wolfgarten M et al. (2013) Interval Debulking Surgery in Patients with Federation of Gynecology and Obstetrics (FIGO) Stage IIIC and IV Ovarian Cancer. Onkologie 36: 324-332. [Crossref]
27. Le T, Alshaikh G, Hopkins L, Faught W, Fung MF (2006) Prognostic significance of postoperative morbidities in patients with advanced epithelial ovarian cancer treated with neoadjuvant chemotherapy and delayed primary surgical debulking. Ann Surg Oncol 13: 1711-1716. [Crossref]
28. Ackroyd SA, Brown J, Houck K, Chu C, Mantia Smaldone G et al. (2018) A preoperative risk score to predict red blood cell transfusion in patients undergoing hysterectomy for ovarian cancer. Am J Obstet Gynecol 219: 598.e1-598.e10. [Crossref]