Findings

The PROSAIC-DS trial finished on 31 July 2023

Summary of findings:

The findings demonstrate the feasibility, safety, effectiveness and potential cost-effectiveness of PROSAIC-DS as an AI clinical decision support system.

  • PROSAIC-DS can consistently and accurately triage up to 33% of prostate cancer patients away from the MDT (over 96% agreement with an MDT decision);
  • The tool was successfully integrated with existing IT systems;
  • Qualitative findings suggest PROSAIC-DS improves the care experience and empowers patients to make better-informed choices;
  • The research was inconclusive about whether PROSAIC-DS increased MDT compliance with evidence based best practice;
  • The economic evaluation was not completed, but reducing the MDT list by 33% would likely generate considerable opportunity cost/efficiency savings by releasing MDT clinicians for other clinical work. Patient outcomes could be improved by reducing the referral to treatment time by 1-2 weeks, and allowing time for the MDT to discuss complex cases.

Following the successful results, discussions are underway to implement the tool in other MDTs at Guy’s Hospital (58), and at other Trusts and Cancer Alliances.

 

Detailed findings:

1. Retrospective analysis

A total of 287 consecutive historic new referrals to KCH were identified in phase one, and information on the historic MDT outcome was available for 152 patients. Overall concordance between the clinicians’/panel recommendations and PROSAIC-DS was found to be very high at 92% (CI 88.1%-94.7%). Concordance between the clinicians’/panel recommendations and the historic MDT was much lower at only 53% (CI 47% – 59%).

The C5.0 machine learning algorithm generated a decision tree that triages patients to PROSAIC-DS for streamlining if:

Life expectancy > 10yrs AND Cancer Stage = I, IIA, IIIc, IVb

OR

Life expectancy > 10yrs AND ISUP Grade III AND Cancer stage = Iic, IIIa, IIIb, Iva

Applying this triage decision tree to the retrospective cohort of KCH patients resulted in 45% of patients being streamlined to PROSAIC-DS, and for this sub-set of patients PROSAIC-DS’ treatment recommendations were 100% concordant with the clinicians’/panel recommendations.

2. Triage

416 prospective patients referred to Guy’s hospital were identified in phase two. Of these, 61 were excluded from the overall concordance analysis because no data were available. In line with an intention to treat analysis, we included all 416 patients in our denominator for the triage analysis; however, we still had to exclude patients where no MDT recommendations were available.

For the 355 cases where data was available, overall concordance between the MDT recommendations and PROSAIC-DS was found to be moderately high at 85.6%. 

Using the machine learning derived decision tree to ‘virtually’ triage patients for streamlining by PROSAIC-DS, it was possible to assign 93 (22.3%) of 416 patients to ‘not for discussion at MDT’. For this sub-group of patients, for 91 patients where MDT results were available, concordance between MDT recommendations and PROSAIC-DS recommendations was 97.8%. When the clinically derived decision tree was used, it was possible to assign 141 (33.8%) of 416 patients to ‘not for discussion at MDT’, and for this larger sub-group, for 138 patients where MDT results were available, a marginal reduction in concordance was observed, with 96.3% concordance between MDT recommendations and PROSAIC-DS recommendations.

It was not possible to implement live triage within the time period of the study. However, the Guy’s Prostate MDT has developed and approved a streamlining protocol and agreed standards of care (SoC) for less complex patients that incorporates PROSAIC-DS. PROSAIC-DS is currently being re-programmed to incorporate the clinical triage decision criteria. This will allow PROSAIC-DS to: automatically identify which patients are eligible for streamlining; will make treatment recommendations in accordance with the agreed SoC; and automatically export these decisions into the EPR. The plan is to implement live automated streamlining in the next 6-8 weeks. Outside of this NIHR study, we have already completed a retrospective validation of Deontics CDSS with our breast cancer MDT.

The Deontics CDSS was able to achieve even higher performance, triaging 78% of breast cancer patients to ‘not for discussion at MDT’ with 100% concordance between the Deontics CDSS and best practice.

3. RCT

We were able to contact 225 patients who had been referred to the GSTT MDT from Feb-June 2023. Of these, 161 (71.5%) consented and 64 declined to take part. Due to industrial action, it was not possible to use PROSAIC-DS in the MDT on two occasions, and IT issues precluded its use on another occasion, resulting in the loss of 24 patients. For a further 7 patients there was no record of the MDT decision on the system; thus 130 patients were successfully randomised to PROSAIC-DS recommendations available/ No PROSAIC-DS recommendations available.

The primary outcome was concordance between the MDT and best practice guidance. Where there was disagreement between the MDT recommendations and PROSAIC-DS, these cases were reviewed by an independent panel of a urologist and clinical oncologist to determine best practice; if the panel deemed PROSAIC-DS’ recommendation to be best practice, then PROSAIC-DS was reclassified as concordant; this occurred for 3 patients.

In the intervention group, the MDT complied with best practice for for 52/57 [91.23%, (95% confidence interval: 83.88%, 98.57)] patients; for the control group the MDT complied with best practice for 65/73 [89.04%, (95% C: 81.88%, 96.21%)] patients. The percent difference of 2.19% (95% C: -8.07%, 12.45%); p= 0.680) was not statistically significant at the 5% nor at the 10% level of significance. It has been unfortunate that we were not able to achieve a sample size of sufficient statistical power to adequately evaluate the value of PROSAIC-DS within the MDT.

4. Qualitative study – useability/acceptability by clinicians

The study explored both positive and negative views of the tool by the clinicians that used it. Positively, the tool was praised for streamlining complex decision-making processes and generating evidence based recommendations that enhance trust among clinicians. It expanded treatment options by introducing previously overlooked approaches and facilitates discussions by focusing attention on key points. For specialist MDTs, the tool provided consistent, evidence-driven recommendations that reinforced decision-making.

However, concerns were voiced about the tool’s user interface, complexity, and its potential to add time to already rapid MDT discussions. Resistance to change was observed among clinicians accustomed to conventional methods, and the challenge of integrating the tool seamlessly into existing workflows was highlighted. There was also uncertainty about its assimilation, particularly in the face of upcoming changes (with hospital EPRs), prompting a call for user-centred design.

The study found that while the PROSAIC-DS tool offers immense potential, addressing user interface concerns, mitigating resistance to change, and ensuring smooth integration are essential.

5. Patient Decision Aid

The interviews conducted to explore the impact of the PDA on treatment decision-making for patients revealed a range of positive outcomes, highlighting the aid’s effectiveness in this context.

The PDA played a pivotal role in equipping participants with a comprehensive understanding of the available treatment options. By presenting detailed information about the benefits and potential risks associated with each option, the PDA enabled patients to make more informed choices. This aspect was particularly appreciated by participants, as it empowered them to enter their consultations with a heightened sense of confidence and preparedness.

One of the remarkable findings was the PDA’s ability to foster active involvement from patients in the decision making process. By prompting them to evaluate their personal values and preferences, the tool facilitated a patient centred approach. This shift was significant, as participants expressed a greater sense of control over their treatment journeys, ensuring that their decisions aligned with their individual circumstances and desires.

An area where the PDA demonstrated substantial positive impact was in reducing the levels of anxiety and uncertainty experienced by patients. Through its clear explanations and visual aids, the tool demystified the complexities of various treatment options. Consequently, participants reported feeling more at ease and better equipped to anticipate the implications of their choices.

Beyond its individual impact, the PDA also emerged as a catalyst for family involvement. Participants who discussed the PDA materials with their families found that it sparked meaningful conversations and provided a platform for their loved ones to contribute to the decision-making process. This familial collaboration not only enhanced the quality of decisions but also strengthened the emotional support network for patients.

The PDA’s recognition of the importance of time and support resources was another key positive aspect. By emphasising the value of taking the necessary time and involving family members, the tool validated participants’ need for a well-considered and holistic approach to decision-making.

Furthermore, the PDA positively influenced patient-clinician interactions. Participants armed with comprehensive knowledge from the tool reported feeling more confident in engaging with their healthcare providers. This enhanced communication, facilitated more productive discussions during consultations, where participants could seek clarification on specific points and express their preferences more effectively.

Overall, the analysis of the interviews underscores the PDA’s pivotal role in the treatment decision-making process for patients. Its ability to provide comprehensive information, promote patient engagement, alleviate anxiety, foster family collaboration, validate the significance of time, and improve patient-clinician communication collectively contribute to a positive and patient-centric decision-making experience. The PDA emerges not only as an information provider but also as an empowering tool that enriches the overall quality of treatment decisions, promoting a more confident and well-supported patient journey.

6. Cost effectiveness

A comprehensive cost effectiveness analysis was not possible within the study as relevant results were only available at the end of the study period. KiTEC has agreed to undertake this work post-study with the intention of publishing results in 6-12 months. The following results were observed: the RCT and Patient Decision Aid use cases were not sufficiently developed to derive cost benefits (although in future they could be). However, the triage use case has created opportunity benefits due to the freeing up of clinician time, and potentially moving patients on the treatment pathway a week or two earlier. Guy’s has the intention of replacing the Local MDT with PROSAIC-DS. This MDT runs for 30 mins and on average reviews 15 patients. In attendance is generally:

  • 1 Clinical Nurse Specialist
  • 1 research person
  • 1 Cancer Data Team member
  • 2 urology registrars
  • 1 oncology registrar
  • 1 radiologist
  • 1 histopathology
  • 2 urology consultants
  • 2 oncology consultants

The Specialist MDT has double the amount of patients and more clinicians in attendance and lasts around 70 minutes, so the potential of triaging up to 33% of patients away from this MDT could reduce the time by 23 minutes.

 

Conclusions

Our findings demonstrate the feasibility, safety, effectiveness and potential cost-effectiveness of PROSAIC-DS as an AI CDSS that can automate the streamlining of a prostate cancer MDT. This process includes the automation of: the selection of patients eligible (on pre-agreed clinical criteria) for triage to an agreed standard of care (SoC); the assignment of those patients to an appropriate standard of care in line with local, national and international evidence based guidelines; the recording of the recommended treatment in the patient’s electronic patient record; and the listing of the patient as ‘not for discussion’ at the MDT.

Our results show that PROSAIC-DS is able to consistently triage as many as 33% of prostate cancer patients to ‘not for discussion at MDT’ and assign them to the appropriate SoC with a very high level of accuracy (over 96% agreement with an MDT decision), deemed sufficient to permit automated streamlining, provided that the treating clinician reviews PROSAIC-DS’ recommendations, and has the option to ignore them and/or refer the patient back to the MDT.

Because of the sample size of the RCT component of the study, and the failure to fully engage MDT clinicians in using PROSAIC-DS in the MDT, it was not possible to draw any conclusions about the impact of PROSAIC-DS in increasing MDT compliance with evidence based best practice.

We were not able to complete the economic evaluation, but it is not unreasonable to conclude that an average reduction of the MDT list by 33% would generate considerable opportunity cost/efficiency savings by releasing MDT clinicians to spend more time: seeing and treating patients on the wards or in outpatients; in the operating theatre; or reading MRIs and histopathology slides. Streamlining the MDT may also improve patient outcomes by reducing the referral to treatment time by as much as 1-2 weeks, and allowing more time in the MDT to discuss more complex cases.

Our qualitative findings suggest that the PROSAIC-DS patient decision aid, by providing personalised and curated information on the outcomes and adverse effects of treatments, and by eliciting their preferences for different outcomes and for avoiding adverse effects, improves the care experience and empowers patients to make more informed choices.