We reported this retrospective cohort study following the suggestion of the TRIPOD [14] (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis Statement), TRIPOD-AI [15] (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis-Artificial Intelligence Statement), and Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research [16] (Multimedia Appendix 1). The framework of the study is shown in Figure 1.

Our research team included multidisciplinary experts in the field of epidemiology, pathology, ultrasound, surgery, and endocrinology. We conducted a retrospective cohort study at Peking University Third Hospital in Beijing, China, from January 2012 to September 2023. The primary data source for this retrospective cohort study was the electronic medical record system of Peking University Third Hospital. Patients were included if they were pathologically diagnosed with FTC or FTA after surgical treatment; patients were excluded if they did not have ultrasound examinations before surgery. To ensure the accuracy of the pathological diagnosis of FTN, we invited the experienced pathologists to double-check all the pathological diagnoses of FTN based on the 2022, 5th edition WHO Classification of Thyroid Neoplasm [17].

We first scanned the entire dataset to identify and remove duplicate patient records by using a combination of unique identifiers such as patient ID, date of surgery, and specific pathological report numbers. For categorical variables with a relatively low proportion (5% or less) of missing values, we used imputation based on the majority class. For numerical variables such as patient age or tumor size, we used mean imputation. To classify the FTNs into small- and large-sized categories, we developed a restricted cubic spline model to identify the cutoff value. Specifically, we used the maximum tumor diameter as the continuous variable (predictor); the malignancy risk of FTNs as the outcome; and the covariates included composition, echogenicity, margin, halo, taller-than-wide, calcification, internal blood flow, vascularity, trabecular formation, nodule-in-nodule appearance, mean diameter, thyroid-stimulating hormone (TSH), free triiodothyronine (FT3), free thyroxine (FT4), mean TSH score (interval-adjusted detailed TSH score [18]), tRMSSD of TSH (the time-adjusted root mean square of successive differences of TSH [19]), mean TSH (mean value of preoperative TSH), and coefficient of variation of TSH (the ratio of the SD of preoperative TSH to the mean value of preoperative TSH). To determine the optimal number of nodes for the restricted cubic splines, we employed the Akaike information criterion to strike the balance between the model goodness of fit and complexity that most effectively aligns with the data [20]. Finally, we divided FTNs into two groups: small- (maximum diameter less than the cutoff value) and large-sized (maximum diameter greater than the cutoff value) FTNs.

We selected the predictors for the malignancy risk of FTNs based on our domain knowledge [21,22] and data available. The predictors mainly included patients’ gender, age, BMI, ultrasound features, thyroid hormones, and Hashimoto thyroiditis. To ensure the validity of measurements of predictors, both researchers and clinicians carefully checked the data source of the predictors.

Ultrasound features included composition (solid, predominantly solid, predominantly cystic, or cystic), echogenicity (hyperechoic, isoechoic, hypoechoic, or anechoric), margin (circumscribed, ill-defined, irregular, or lobulated), halo (absent halo, even thickness halo, present halo without evenness of thickness reported, or uneven thickness halo), taller-than-wide (absent or present), calcification (no echogenic foci, microcalcification, macrocalcification, peripheral calcification, microcalcification with comet-tail artifacts, or punctate echogenic foci of undetermined significance), internal blood flow (absent or present), vascularity (mainly central vascularity, mainly peripheral vascularity, mixed vascularity, or avascularity), trabecular formation (absent or present), nodule-in-nodule appearance (absent or present), and mean diameter.

The measurements of thyroid hormones included TSH, FT3, FT4, and TSH-related features. As listed in Table S1 in Multimedia Appendix 2, TSH-related features included the mean TSH score, the tRMSSD of TSH, mean TSH, and coefficient of variation of TSH. The diagnostic criteria for Hashimoto thyroiditis referred to the ultrasonography describing the thyroid tissue as substantial diffuse lesions and the pathology report describing the thyroid tissue as Hashimoto thyroiditis.

We established the machine learning–based models, as shown in Figure 1. We trained eight classification models including logistic regression, lasso regression, weighted k-nearest neighbor, decision tree, random forest, naive bayes, XGBoost, and support vector machine (SVM). In total, 70% of the dataset was allocated for model training and selection, while the remaining 30% was reserved for internal validation. We developed and validated the models stepwise through predictor preprocessing, model training, hyperparameter tuning, and 5-fold cross-validation. It is important to note that our study population was from the real-world, naturally distributed population so that the outcome variable was slightly imbalanced (approximately 30% was FTC among all types of FTNs). We adopted the synthetic minority over-sampling technique (SMOTE), which increased the sample size of a few classes by creating new synthetic samples rather than simply copying existing ones [23]. We evaluated the model performance using accuracy, F1 score, the area under the receiver operating characteristic curve (AUROC), the area under the precision-recall curve (AUPRC), sensitivity, and specificity. We comprehensively considered the performance and interpretability of models and selected the most suitable model. We used the mlr3 [24] ecosystem in R 4.4.1 to conduct machine learning.

We selected important predictors for small- and large-sized FTNs, respectively. We evaluated the importance of predictors (feature importance) by computing the cross-entropy loss (loss: ce) of all features and visualized the importance of features. We identified the first five most important predictors that could both predict the outcome and not overlap with other predictors based on medical expertise and a novel information-gain approach. Based on the concept of entropy from information theory, the information gain approach is used to assess the extent to which features reduce uncertainty or increase the amount of information [25]. This approach helps to determine which features most effectively enhance classification accuracy by calculating the difference in information entropy before and after feature classification [25].‌ After the selection of important predictors, we calculated the odds ratio (OR) with 95% CI and drew the forest maps using multivariate logistic regression models.

This study was approved by the Medical Research Ethics Committee of Peking University Third Hospital (No. IRB00006761- M2023168). As a retrospective analysis, the study was granted a waiver for additional informed consent. During the data extraction process, strict confidentiality measures were implemented to ensure patient privacy and data security. All extracted data were anonymized, with any information that could directly identify patients being removed.

Among the included 1494 patients, 1266 (84.7%) were diagnosed with FTA, and the remaining were diagnosed with FTC; the average (SD) age of the patients was 48.25 (0.75) years, and 1127/1494 patients (75.4%) were female. We used a restricted cubic spline model with three nodes that optimally balanced fitting the nonlinear relationship within the data while minimizing the overfitting risk. This analysis revealed a key turning point: the slope of the curve changed distinctly at a maximum diameter of 3 cm for FTNs (Figure 2), indicating that the influence of the maximum diameter on the risk of FTN malignancy shifted at this threshold. As such, 715/1494 tumors (47.9%) were classified as small-sized FTNs (maximum diameter <3 cm), while 779/1494 tumors (52.1%) were large-sized FTNs (maximum diameter ≥3 cm). The characteristics of the study population and FTNs are listed in Tables 1 and 2, respectively.

We compared the performance in discrimination among the eight models (logistic regression, weighted k-nearest neighbor, lasso regression, decision tree, random forest, naive bayes, XGBoost, and SVM). The XGBoost and random forest models performed broadly better in the small- and large-sized FTN groups, respectively (Table 3, Table S2 in Multimedia Appendix 2).

In the small-sized FTN group, the top five predictors were the mean TSH score, tRMSSD of TSH, age at hospital admission, mean diameter, and calcification in the XGBoost model (Figure 3). Compared to tumors with no echogenic foci in the ultrasound image, the small-sized FTNs expressing microcalcification (OR 2.10, 95% CI 0.98-4.30), macrocalcification (OR 2.90, 95% CI 1.50-5.60), peripheral calcification (OR 4.50, 95% CI 1.50-13.00), and microcalcification with comet-tail artifacts (OR 1.60, 95% CI 0.09-9.50) had a higher risk of malignancy. Additionally, the risk of malignancy was higher in the patients with small-sized FTNs with a lower mean TSH score (OR 1.49, 95% CI 1.20-1.85), lower tRMSSD of TSH (OR 1.03, 95% CI 0.83-1.72), younger patients (OR 1.33, 95% CI 1.05-1.69), and larger mean diameter (OR 1.40, 95% CI 1.10-1.70).

By contrast, in the large-sized FTN group, the top 5 predictors were the mean TSH score, tRMSSD of TSH, BMI, nodule-in-nodule appearance, and mean diameter in the random forest model (Figure 3). The risk of malignancy was higher in the large-sized FTN, which included the lower mean TSH score (OR 1.61, 95% CI 1.37-1.96), lower tRMSSD of TSH (OR 1.11, 95% CI 0.91-1.49), higher BMI (OR 1.20, 95% CI 0.97-1.40), the presence of nodule-in-nodule (OR 3.30, 95% CI 1.30-7.90), and larger mean diameter (OR 1.50, 95% CI 1.20-1.70).

Taking advantage of a long-term cohort based on the real-world data, we found distinctive predictors for the malignancy risk of FTNs between the small- and large-sized tumors. Specifically, the tumor’s calcification appearance, mean diameter, and patients’ age were more important in predicting the malignancy risk of small-sized FTNs, whereas the tumor’s nodule-in-nodule appearance and patients’ BMI were more important in that of large-sized FTNs.

It remained unclear regarding the cut-off value of tumor size that indicated the malignancy risk or surgical indications of FTNs. Concerning clinical guidelines, we observed that neither domestic nor international guidelines have recommended a cut-off value of tumor size for surgical treatment in FTNs [26-28]. Regarding existing research, among the 5 studies included in this review, the cut-off value of tumor size used to predict the malignancy risk of FTNs remained contradictor[29-33]. The included studies reported the associations between tumor size and the malignancy risk of FTNs, but all of them lacked a solid basis for their tumor-size classification criteria prior to surgical treatment.

Based on our literature review, previous studies on this topic often employed a one-size-fits-all model to predict the malignancy risk of FTN, without considering the tumor size[34,35]. For example, a retrospective multicenter study developed a prediction model for FTNs using the Thyroid Imaging Reporting and Data System (TI-RADS), and the validation dataset showed that the follicular TI-RADS model improved the performance in differentiating between FTA and FTC (AUROC 0.81) [34]. Additionally, a model combining the prior-based level set method with a deep convolutional neural network achieved an AUROC of 0.913 in distinguishing FTC from FTA using features extracted from ultrasound images [35]. Furthermore, we evaluated the follicular TI-RADS scoring criteria developed by Li et al [34] with our data and found a lower sensitivity (0.044 with a threshold for FTC risk set at >90%, and 0.269 with using a >50% FTC risk threshold) in evaluating the prediction model. These inconsistencies and the reduced sensitivity observed in external datasets may, at least in part, be attributed to the limitation of not considering tumor size when predicting the malignancy risk.

Our findings suggest that clinical differentiation between benign and malignant FTNs should be more precisely tailored according to the size of the FTNs. For example, the nodule-in-nodule appearance, a characteristic feature of FTC, was more frequently observed in larger FTNs [36]. This sign may reflect heterogeneous tumor cell proliferation, a phenomenon more commonly associated with larger tumors. Conversely, the incidence of this sign is relatively low in small-sized FTNs, complicating the identification of small-sized FTCs. Calcification, however, demonstrates a higher predictive value for malignancy in smaller FTNs. Our study focused on commonly used and easily obtainable predictors in clinical practice, including clinical features, TSH levels, and ultrasound characteristics. The limited number of predictors and the straightforward model design enhance its practicality for physicians. This research assists thyroid specialists in customizing predictor selection to assess the malignancy risk in FTNs of different sizes, ultimately improving the accuracy of FTC identification, patient outcomes, and quality of life while reducing postoperative complications.

Our finding also suggests that the mean TSH score ranked among the top 5 predictors for determining malignancy in tumors of both sizes. A cohort study from the EPIC (European Prospective Investigation into Cancer and Nutrition) cohort has revealed a negative association between elevated TSH levels and an increased risk of thyroid cancer [37]. Similarly, Gudmundsson et al [38] propose that low TSH levels may reduce the differentiation of thyroid epithelium, potentially increasing the predisposition to malignant cell transformation.

In evaluating the predictive performance of our machine learning models, we employed metrics distinct from those used in prior studies. For instance, Li et al [34] reported an AUROC of 0.76 for the LASSO regression model (the ratio of FTA to FTC: training set, 2.74; validation set, 3.70), and also in the LASSO regression model, the AUROC reached 0.913 for discriminating FTA from FTC [35] (the ratio of FTA to FTC: 4.00). However, neither study incorporated the F1 score or AUPRC as evaluation metrics. Given the imbalanced nature of our data (FTA to FTC ratio: 7.00 for small follicular tumors and 5.00 for large follicular tumors), AUPRC provides a more informative and intuitive measure of model performance compared to AUROC [39]. Additionally, the F1 score offers a comprehensive evaluation by integrating precision and recall, ensuring a balanced assessment of model performance in the context of skewed category distributions [40].

Our study had several strengths. It was among the first to classify FTNs into two subgroups based on tumor diameter for machine learning analysis, revealing significant differences between these subgroups. Additionally, our models benefit from a large sample size, the use of clinically accessible and validated predictors, and a comprehensive evaluation using metrics appropriate for imbalanced data [41]. Importantly, the distribution of FTA and FTC in our study population was fully consistent with that of patients with FTNs in real-world settings, avoiding any exaggeration of sample sizes during model training.

However, our study had certain limitations. First, our classification of small- and large-sized FTNs based on a threshold of 3 cm was determined by our dataset rather than established guideline consensus, which may limit its generalizability to external datasets. We recommend that future clinical guidelines refine specific ultrasound risk indicators for FTNs based on the findings of this study and subsequent related research. Second, while we applied the SMOTE oversampling strategy to address imbalanced data, this approach may introduce bias when predicting new data [42]. Finally, the lack of external validation for our trained model limits our ability to assess its generalizability, potentially affecting its practical applicability and reliability in predicting new cases [43].

Our study had important clinical implications. In current clinical practice, it primarily relies on the ultrasonographic features of FTNs to assess the tumors’ malignancy risk under the observation period (ie, before surgical treatment). It is thus crucial to emphasize the need for personalized predictive models in FTNs. Achieving this requires a more refined stratification of tumors based on the clinical characteristics of FTNs, which can enhance the customization of predictive models for tumors of individual cases. This tailored approach acknowledges the inherent variability within FTN categories, suggesting that further stratification within FTNs based on tumor size can facilitate more precise diagnostic and treatment decisions.

To advance this research, we intend to conduct a prospective study to validate the accuracy and reliability of the identified predictors in real-world clinical settings. The prospective evidence will enable updates and refinements to existing medical knowledge and practices for managing FTNs. For FTNs with tumor diameters <3 cm, surgical decision-making should prioritize TSH levels, age at admission, tumor diameter, and calcification status. For FTNs ≥3 cm, TSH levels, BMI, nodule-in-nodule architecture, and tumor diameter should guide clinical recommendations. Furthermore, we advocate that professional medical societies and health care organizations collaboratively develop evidence-based practice guidelines integrating these predictors.

In our study, we identified differences in predictors among follicular tumors of varying sizes. Clinically, these findings emphasize the importance of considering the size during the preoperative diagnosis of benign versus malignant FTNs. We found that both clinical guidelines and the existing research literature have not sufficiently addressed the optimal size of FTNs for surgical intervention or its correlation with malignancy risk. There is a significant research gap in precisely determining the preoperative size-based classification of FTNs. Thus, further investigations are imperative to address this knowledge deficit.