Value of a deep learning-based algorithm for detecting Lung-RADS category 4 nodules on chest radiographs in a health checkup population: estimation of the sample size for a randomized controlled trial

Abstract

Objective: To explore the value of a deep learning-based algorithm in detecting Lung CT Screening Reporting and Data System category 4 nodules on chest radiographs from an asymptomatic health checkup population. Methods: Data from an annual retrospective cohort of individuals who underwent chest radiographs for health checkup purposes and chest CT scanning within 3 months were collected. Among 3073 individuals, 118 with category 4 nodules on CT were selected. A reader performance test was performed using those 118 radiographs and randomly selected 51 individuals without any nodules. Four radiologists independently evaluated the radiographs without and with the results of the algorithm; and sensitivities/specificities were compared. The sample size needed to confirm the difference in detection rates was calculated, i.e., the number of true-positive radiographs divided by the total number of radiographs. Results: The sensitivity of the radiologists substantially increased aided by the algorithm (38.8% [183/472] to 45.1% [213/472]; p <.001) without significant change in specificity (94.1% [192/204] vs. 92.2% [188/204]; p =.22). Pooled radiologists detected more nodules with the algorithm (32.0% [156/488] vs. 38.9% [190/488]; p <.001), without alteration of false-positive rates (0.09 [62/676], both). Pooled detection rates for the annual cohort were 1.49% (183/12,292) and 1.73% (213/12,292) without and with the algorithm, respectively. A sample size of 41,776 in each arm would be required to demonstrate significant detection rate difference with < 5% type I error and > 80% power. Conclusion: Although readers substantially increased sensitivity in detecting nodules on chest radiographs from a health checkup population aided by the algorithm, detection rate difference was only 0.24%, requiring a sample size >80,000 for a randomized controlled trial. Key Points: • Aided by a deep learning algorithm, pooled radiologists improved their sensitivity in detecting Lung-RADS category 4 nodules on chest radiographs from a health checkup population (38.8% [183/472] to 45.1% [213/472]; p <.001), without increasing false-positive rate. • The prevalence of the Lung-RADS category 4 nodules was 3.8% (118/3073) on the population, resulting in only 0.24% increase of the detection rate for the radiologists with assistance of the algorithm. • To confirm the significant detection rate increase by a randomized controlled trial, a sample size of 84,000 would be required. © 2021, European Society of Radiology.