Accuracy of a Single Versus Multiple Trials of Novel Pacemaker ID Algorithm Mobile Phone App for Identification of Cardiac Devices

A B S T R A C T

Fast and accurate identification of cardiac devices can facilitate device programming and interrogation in various medical settings. We have previously demonstrated the accuracy of the PacemakerID machine learning algorithm for mobile phone cardiac device identification. However, the questions of the reproducibility of this algorithm and whether a single trial sufficiently maximizes accuracy have yet to be answered. Here, we examine 502 chest x-rays performed at a single institution on patients with implantable cardioverter-defibrillators and permanent pacemakers. The PacemakerID mobile phone application was used for five sequential trials on each image and the accuracy of one, three, and five trials were compared. A single trial resulted in a 79% accuracy and 82% positive predictive value with no significant difference (p=0.69) as compared to five trials at identifying device manufacturers. Across all devices, the results of a single trial were not significantly different from those of five trials. Our data demonstrate that a single trial is sufficient to maximize diagnostic accuracy with the PacemakerID mobile phone application, facilitating rapid identification for prompt programming and interrogation of cardiac devices.

Keywords

Machine learning, cardiac devices, PacemakerID

Introduction

Rapid and accurate identification of cardiac devices is critical across a variety of healthcare settings to facilitate prompt device interrogation and programming [1]. Although algorithms such as CARDIA-X exist for a cardiovascular implantable electronic device (CIED) manufacturer identification from chest x-ray imaging, they are potentially time-consuming and operator-dependent [2]. We previously described the PacemakerID machine learning algorithm (PIDa) mobile phone application, showing that it provides equivalent to superior accuracy compared to other algorithms [3]. PIDa is available for free, requires only a mobile phone, and identifies manufacturers of both permanent pacemakers (PPMs) and implantable cardioverter-defibrillators (ICDs). User feedback has suggested differing predictions with several trials on the same device, which would limit the ease of use and potential time savings of the app. As the reproducibility of CIED identification via PIDa has not been established, we sought to objectively compare multiple trials with a single use to objectively determine the reproducibility of the algorithm.

Methods

The PIDa mobile phone application for Apple and Android was developed using chest X-rays of patients with CIEDs performed between 2016 and 2018 at a single institution in the United States. In total, 1509 radiographs were included in the analysis. During the image capture process, images had lighting conditions altered, were flipped horizontally and vertically, were subject to random cropping, and had differing levels of contrast and brightness to simulate inter-operator and inter-device variability. Images were assigned via a 7:2:1 ratio to be used for training, validation, and testing, respectively, to generate the algorithm. The algorithm assigns a prediction certainty for four device manufacturers (Biotronik, Boston Scientific, Medtronic, and St Jude Medical [now Abbott]) and unknown [4].

We obtained chest x-rays retrospectively. These were of patients who either underwent CIED initial implant or replacement between January 2019 and October 2019 or of patients who were seen as an inpatient by the electrophysiology service for device interrogation between July and November 2019. Device identity was determined by device interrogation or from the operative note from the time of device placement. Only permanent pacemakers (PPMs) and implantable cardioverter-defibrillators (ICDs) were included.

In this study, PIDa was used for five sequential trials on JPEG images of 502 x-rays with a prediction certainty cutoff of 50%. The accuracy of PIDa in device identification with the first trial, the first three trials, and all five trials was calculated. When repeated trials of the same image resulted in PIDa identifying differing CIEDs, the most frequently identified CIED was selected.

The accuracy of a single trial was compared to that of three trials, five trials, or chance. The outcome was the identification of the correct device, with a Chi-squared test with a p-value of 0.05 considered significant. Additionally, the reproducibility of device identification was determined by comparing the results of one versus five trials by one-tailed paired t-test with a p-value of 0.05 was considered significant.

Results

Of the 502 subjects included, 227 had PPMs and 275 ICDs. The overall accuracy was 79.2%, 79.2%, and 81.2% for one, three, and five trials, respectively (Table 1). A single trial was similar to three and five trials in identifying all devices (p-values = 0.91 and 0.69, respectively). Paired t-tests to evaluate the reproducibility of a single trial versus the mode of five trials showed no significant difference in results (p=0.052). A single trial was significantly superior to chance at device identification (p<0.001).

Table 1: The overall accuracy of PIDa in combined PPM and ICD identification by manufacturer.

Manufacturer	Device Type	Total Number of Devices (n)	One trial	Mode of Three Trials	Mode of Five Trials
Biotronik	All devices	143	120 (83.9%)	125 (87.4%)	125 (87.4%)
Boston Scientific	All devices	82	67 (81.7%)	69 (84.1%)	69 (84.1%)
Medtronic	All devices	162	132 (81.5%)	126 (77.8%)	126 (77.8%
St. Jude Medical (Abbott)	All devices	115	79 (68.7%)	78 (67.8%)	88 (76.5%)
Total	All devices	502	398 (79.2%)	398 (79.2%)	408 (81.2%)

The accuracy of PIDa in the identification of PPMs across all device manufacturers was 78.0%, 76.7%, and 80.6% for one, three, and five trials, respectively (Table 2). Neither three nor five trials were significantly superior to a single trial in identifying PPMs (p-value = 0.91 and 0.84, respectively). Paired t-test to evaluate the reproducibility of a single trial versus the mode of five trials showed no significant difference in results (p=0.07). A single trial was significantly superior to chance at device identification (p<0.001).

Table 2: Accuracy of PIDa in PPM identification by manufacturer.

Manufacturer	Device Type	Total Number of Devices (n)	One trial	Mode of Three Trials	Mode of Five Trials
Biotronik	PPM	54	44 (81.5%)	46 (85.2%)	47 (88.0%)
Boston Scientific	PPM	29	25 (86.2%)	26 (89.7%)	27 (93.1%)
Medtronic	PPM	88	70 (79.5%)	65 (73.9%)	67 (76.1%)
St. Jude Medical (Abbott)	PPM	56	38 (67.9%)	37 (66.1%)	42 (75.0%)
Total	PPM	227	177 (78.0%)	174 (76.7%)	183 (80.6%)

The accuracy of PIDa in ICD identification across all device manufacturers was 80.4%, 81.5%, and 81.5% for one, three, and five trials, respectively (Table 3). Neither three nor five trials were significantly superior to a single trial in identifying ICDs (p-value = 0.98 and 0.86, respectively). Paired t-test to evaluate the reproducibility of a single trial versus the mode of five trials showed no significant difference in results (p=0.20). A single trial was significantly superior to chance at device identification (p<0.001).

Table 3: Accuracy of PIDa in ICD identification by manufacturer.

Manufacturer	Device Type	Total Number of Devices (n)	One trial	Mode of Three Trials	Mode of Five Trials
Biotronik	ICD	89	76 (85.4%)	79 (88.8%)	78 (87.6%)
Boston Scientific	ICD	53	42 (79.2%)	43 (81.1%)	42 (79.2%)
Medtronic	ICD	74	62 (83.8%)	61 (82.4%)	59 (79.7%)
St. Jude Medical (Abbott)	ICD	59	41 (69.5%)	41 (69.5%)	46 (78.0%)
Total	ICD	275	221 (80.4%)	224 (81.5%)	225 (81.5%)

The positive predictive value of identification as a Boston Scientific device was 90.5%, 73.6% for identification as a Biotronik device, 81.0% for identification as a Medtronic device, and 90.8% for identification as a St. Jude Medical device. Across all devices, the positive predictive value was 81.7%. Biotronik and Boston Scientific devices were most frequently misidentified as Medtronic devices while Medtronic and St. Jude Medical devices were most frequently misidentified as Biotronik devices (Figure 1).

Discussion

PIDa was accurate in identifying cardiac devices with a single trial, with a 79.2% accuracy for all device types, and with an overall accuracy of 78% in identifying ICDs and 80% in identifying PPMs. Importantly, accuracy was not significantly increased by using multiple trials. That a single trial is sufficient for an accurate prediction supports the usability and potential time savings of the PacemakerID app.

Despite the high level of accuracy, we demonstrated here, previous studies have demonstrated a range of accuracies of 60.5-89% with PIDa [3-7]. Variability is likely the result of intra-operator and intra-phone variability, with one study demonstrating significantly reduced accuracy of 80% versus 86% when the app was used off-axis [5]. Based on our experience, variability in camera quality, quality of chest x-ray studies, and lighting may be other underlying variables influencing accuracy. Despite these potential sources of error, overall accuracy in identifying CIEDs was high, and the positive predictive value of device identification was 81.7%.

In addition to being sufficiently accurate for practical use, we also demonstrated that PIDa gave highly reproducible results when trialed multiple times using the same image with the same user. There were no significant differences between results obtained from a single trial and the mode of three or five trials when stratified by ICDs, PPMs, or for all devices by paired t-test. Thus, despite the variability in accuracy depending on the factors listed above, a single trial is both sufficient to maximize accuracy and putative device identification is unlikely to change with multiple trials. This too enables users to identify devices quickly, with confidence that repeated attempts will not meaningfully change the device identified.

These factors are important as rapid pacemaker identification has a role in many healthcare settings in facilitating pacemaker interrogation [8]. For this reason, other identification methods using chest x-rays such as the CaRDIA-X algorithm, have been proposed [2]. Unlike PacemakerID, these methods require training, are operator dependent, and can be time-consuming.

Further facilitating ease and rapidity of use, PIDa has a simple user interface (Figure 2). It requires only a mobile phone and a free app, and correct use of the app is highly intuitive. Devices are identified by pointing the camera at the device on the chest x-ray and capturing the image. Although there are factors that may lead to inter-operator variability, as mentioned above, the simplicity of use of the app minimizes this. Therefore, this app can be utilized by trainees with differing levels of familiarity with CIEDs.

There are several limitations to this study. Only a limited number of users participated in data collection; thus, although the design of PIDa should minimize operator-dependence, our reported accuracy may be influenced by inter-operator variability. Significantly, phone models and camera quality were variable across PIDa users. This was also a single-center study, limiting our ability to generalize results to other centers that may have a differing frequencies of CIEDs manufactured by different companies. For example, the relatively high level of misidentification of other devices such as Medtronic and Biotronik devices, may be a reflection of the relative greater abundance of those devices amongst in the dataset used to train PIDa. Future updates to this app using data from multiple centers could ameliorate this source of bias.

In conclusion, PIDa represents a promising development in cardiology. Its ease of use and accuracy make it a useful tool in the identification of CIEDs, especially across settings where rapid identification is necessary or where other tools for device identification may not be readily available.

Funding

Funding was provided by the authors.

Conflicts of Interest

Dr. Fisher is a consultant for Medtronic. There has been no communication between Dr. Fisher and Medtronic regarding the present project.

Article Info

Article Type

Research Article

Publication history

Received: Tue 16, Aug 2022
Accepted: Mon 05, Sep 2022
Published: Fri 23, Sep 2022

Copyright

© 2023 Kevin J Ferrick. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Hosting by Science Repository.
DOI: 10.31487/j.JICOA.2022.03.04

Author Info

Alex Conant Jay J Chudow Syona S Shetty Rahul Grover John D Fisher Andrew Krumerman Kevin J Ferrick

Corresponding Author

Kevin J Ferrick
Department of Cardiology, Montefiore Medical Center, Bronx, New York, USA

Figures & Tables

Table 1: The overall accuracy of PIDa in combined PPM and ICD identification by manufacturer.

Manufacturer	Device Type	Total Number of Devices (n)	One trial	Mode of Three Trials	Mode of Five Trials
Biotronik	All devices	143	120 (83.9%)	125 (87.4%)	125 (87.4%)
Boston Scientific	All devices	82	67 (81.7%)	69 (84.1%)	69 (84.1%)
Medtronic	All devices	162	132 (81.5%)	126 (77.8%)	126 (77.8%
St. Jude Medical (Abbott)	All devices	115	79 (68.7%)	78 (67.8%)	88 (76.5%)
Total	All devices	502	398 (79.2%)	398 (79.2%)	408 (81.2%)

Table 2: Accuracy of PIDa in PPM identification by manufacturer.

Manufacturer	Device Type	Total Number of Devices (n)	One trial	Mode of Three Trials	Mode of Five Trials
Biotronik	PPM	54	44 (81.5%)	46 (85.2%)	47 (88.0%)
Boston Scientific	PPM	29	25 (86.2%)	26 (89.7%)	27 (93.1%)
Medtronic	PPM	88	70 (79.5%)	65 (73.9%)	67 (76.1%)
St. Jude Medical (Abbott)	PPM	56	38 (67.9%)	37 (66.1%)	42 (75.0%)
Total	PPM	227	177 (78.0%)	174 (76.7%)	183 (80.6%)

Table 3: Accuracy of PIDa in ICD identification by manufacturer.

Manufacturer	Device Type	Total Number of Devices (n)	One trial	Mode of Three Trials	Mode of Five Trials
Biotronik	ICD	89	76 (85.4%)	79 (88.8%)	78 (87.6%)
Boston Scientific	ICD	53	42 (79.2%)	43 (81.1%)	42 (79.2%)
Medtronic	ICD	74	62 (83.8%)	61 (82.4%)	59 (79.7%)
St. Jude Medical (Abbott)	ICD	59	41 (69.5%)	41 (69.5%)	46 (78.0%)
Total	ICD	275	221 (80.4%)	224 (81.5%)	225 (81.5%)

Figure 1: Percentage of devices by manufacturer identified as each device.

Figure 2: A) Example cardiac device viewed with PIDa and B) capture with device identification.

References

1.     Jerjes Sánchez C, Gonzalez Rayas JM (2019) Pacemaker Emergencies in the ER. In: Cardiology in the ER. Cham: Springer International Publishing 271-305.

2.     Jacob S, Shahzad MA, Maheshwari R, Panaich SS, Aravindhakshan R (2011) Cardiac rhythm device identification algorithm using x-rays: CaRDIA-X. Heart Rhythm 8: 915-922. [Crossref]

3.     Chudow JJ, Jones D, Weinreich M, Zaremski L, Lee S et al. (2021) A Head-to Head Comparison of Machine Learning Algorithms for Identification of Implanted Cardiac Devices. Am J Cardiol 144: 77-82. [Crossref]

4.     Weinreich M, Chudow JJ, Weinreich B, Krumerman T, Nag T et al. (2019) Development of an Artificially Intelligent Mobile Phone Application to Identify Cardiac Devices on Chest Radiography. JACC Clin Electrophysiol 5: 1094-1095. [Crossref]

5.     Sinha SK, Boyle B, Love CJ, Marine JE, Chrispin J et al. (2021) B-PO02-043 Radiographic identification of CIED manufacturer: x-ray logo versus smartphone “Pacemaker-ID” app. Heart Rhythm 18: S113.

6.     Shams P, Mehdi M, Ali J, Ahmed I, Saadia S et al. (2021) Comparing sensitivity and specificity of pacemaker ID application and cardiac rhythm management device-finder application in identifying cardiac implantable electronic device manufacturer using chest radiograph - An observational study. Ann Med Surg 69: 102741. [Crossref]

7.     Sabottke C, Breaux M, Lee R, Foreman A, Spieler B (2021) Analysis of Potential for User Errors in Mobile Deployment of Radiology Deep Learning for Cardiac Rhythm Device Detection. J Digital Imaging 34: 572-580. [Crossref]

8.   McMullan J, Valento M, Attari M, Venkat A (2007) Care of the pacemaker/implantable cardioverter defibrillator patient in the ED. Am J Emerg Med 25: 812-822. [Crossref]