Automated Detection of Steno-Occlusive Lesion on Time-of-Flight MR Angiography: An Observer Performance Study

BACKGROUND AND PURPOSE: Intracranial steno-occlusive lesions are responsible for acute ischemic stroke. However, the clinical benefits of artificial intelligence (AI)-based methods for detecting pathologic lesions in intracranial arteries have not been evaluated. We aimed to validate the clinical utility of an AI model for detecting steno-occlusive lesions in the intracranial arteries. MATERIALS AND METHODS: Overall, 138 TOF-MRA images were collected from 2 institutions, which served as internal (n ¼ 62) and external (n ¼ 76) test sets, respectively. Each study was reviewed by 5 radiologists (2 neuroradiologists and 3 radiology residents) to compare the usage and nonusage of our proposed AI model for TOF-MRA interpretation. They identified the steno-occlusive lesions and recorded their reading time. Observer performance was assessed by using the area under the jackknife free-response receiver operating characteristic curve (AUFROC) and reading time for comparison. RESULTS: The average AUFROC for the 5 radiologists demonstrated an improvement from 0.70 without AI to 0.76 with AI (P ¼ .027). Notably, this improvement was most pronounced among the 3 radiology residents, whose performance metrics increased from 0.68 to 0.76 (P ¼ .002). Despite an increased reading time by using AI, there was no significant change among the readings by radiology residents. Moreover, the use of AI resulted in improved interobserver agreement among the reviewers (the intraclass correlation coefficient increased from 0.734 to 0.752). CONCLUSIONS: Our proposed AI model offers a supportive tool for radiologists, potentially enhancing the accuracy of detecting intracranial steno-occlusion lesions on TOF-MRA. Less experienced readers may benefit the most from this model.