Machine Learning-Based Selection of Resection vs Transplant and Survival in Hepatocellular Carcinoma

Importance: Liver transplantation (LT) generally provides superior long-term survival compared with surgical resection (SR) for hepatocellular carcinoma (HCC), but optimal treatment selection remains challenging due to donor scarcity and patient heterogeneity. Objective: To develop and validate a machine learning (ML)-based decision-support model to estimate optimized individualized treatment selection between LT and SR in HCC. Design, Setting, and Participants: This nationwide cohort study included patients with HCC who underwent either LT or SR between 2008 and 2018 from the Korea Central Cancer Registry as the derivation cohort. An independent cohort of patients with HCC treated between 2009 and 2020 at Seoul St Mary's Hospital was used for external validation. Data were analyzed from February to March 2025. Exposures: Curative treatment with LT or SR for HCC. Main Outcomes and Measures: Separate ML models estimating 3-year overall survival (OS) were developed for LT and SR. Patients were stratified into high- and low-risk groups for each treatment, identifying LT-favorable and LT-nonfavorable groups. Counterfactual analysis evaluated OS differences between ML-guided and clinical practice treatments. Results: A total of 3915 patients (3137 [80.1%] male), 296 in the LT group (median [IQR] age, 54.0 [49.0-60.0] years) and 3619 in the SR group (median [IQR] age, 58.0 [51.0-66.0] years), were included in the derivation cohort, and 614 patients (497 [80.9%] male)-314 in the LT group (median [IQR] age, 55.0 [51.0-60.0] years) and 300 in the SR group (median [IQR] age, 59.0 [52.0-66.0] years)-were included in the external validation cohort. Across both cohorts, LT recipients were generally younger and had more advanced liver disease, with higher rates of cirrhosis (78 [26.4%] vs 699 [19.3%]; P = .005), hepatic encephalopathy (20 [6.8%] vs 10 [0.3%]; P < .001), and ascites (50 [19.9%] vs 153 [4.2%]; P < .001). LT recipients also exhibited poorer liver function, with lower albumin levels (median [IQR], 3.4 [2.8-4.0] vs 4.2 [3.9-4.5] g/dL), higher bilirubin levels (median [IQR], 1.4 [0.9-2.5] vs 0.7 [0.5-1.0] mg/dL), and prolonged international normalized ratios (median [IQR], 1.2 [1.1-1.5] vs 1.1 [1.0-1.1]), and had smaller tumors (median [IQR], 2.3 [1.5-3.6] vs 3.2 [2.2-5.0] cm; P < .001) but more tumors (mean [SD], 1.6 [1.0] vs 1.2 [0.7]; P < .001). The support vector machine model achieved the highest area under the receiver operating characteristic curve (AUROC) of 0.82 (95% CI, 0.78-0.86) in the LT cohort, whereas CatBoost performed best in the SR cohort (AUROC, 0.79 [95% CI, 0.78-0.80]). Counterfactual analysis estimated that ML-guided treatment decisions could improve survival compared with observed clinical practice decisions (HR, 0.46 [95% CI, 0.42-0.50]; P < .001). These findings were consistent in the independent cohort. Conclusions and Relevance: Findings from this cohort study of patients with HCC indicated that an ML-based decision-support model estimated accurate risk stratification and identified the potential for improved survival through individualized, model-guided treatment selection. These findings suggest clinical utility in supplementing existing guidelines.