This study investigated stroke occurrence among PLWH using two datasets for the national (U.S.) and the Tennessee cohorts (Table 1). The national dataset was extracted from the Cerner Health Facts database, and the Tennessee dataset was sourced from the University of Tennessee Health Science Center (UTHSC) Research Enterprise Data Warehouse (rEDW), a standardized, in-house-developed healthcare data repository. The rEDW aggregates data from multiple healthcare organizations across Tennessee, encompassing records for approximately 4.5 million patients. Both datasets comprised individuals with a history of stroke and a control group of those without.

PLWH were identified using ICD-9 code 042 and ICD-10 codes B20-B24. Stroke cases were identified using relevant stroke-related ICD codes across all healthcare encounter types. For cases, laboratory data from before the index stroke visit were extracted. For controls without a history of stroke, a pseudo-index date was assigned based on the most recent stroke visit date of a matched case. Controls were matched on gender, race, and age at the time of HIV diagnosis, and their laboratory data recorded prior to the pseudo-index date were extracted for the analysis. No fixed prediction horizon was imposed; the models discriminate PLWH who experienced a stroke from those who did not, using data recorded before the index or pseudo-index date.

Missing data are common in EHR datasets. To address this issue, we used a cohort-specific missingness threshold to select features for machine learning. For the Tennessee cohort, features with >32% missing values were excluded, yielding 44 features. Given the higher frequency of missing data in the national dataset, a more lenient threshold of ∼77% was used, resulting in ∼41 features. Across cohorts, 28 features were shared; each cohort included 3 additional features selected empirically for performance (Table 2), for a total of 31 features used per cohort. In addition, we performed patient-level filtering within the non-stroke groups: any record with >10% missing values across the retained features was removed. This step reduced the size of the non-stroke cohorts (to 46% of the original size in Tennessee and ∼21% nationally) while retaining all stroke cases to preserve minority-class signal (Table 3).

Two machine learning models were developed and evaluated: a Random Forest model [13, 14] and an XGBoost model [15].

Model performance was assessed using two key metrics:

Area Under the Receiver Operating Characteristic Curve (AUROC): It measures a model’s ability to distinguish between PLWH with stroke and without stroke. It is calculated as the area under a plot with the Y-axis as true positive rate (sensitivity) and the X-axis as the false positive rate across different classification thresholds.

Both the above metrics range from 0 to 1; a value closer to 1 is considered an excellent prediction, while a value closer to 0 is considered the worst in both cases.

To ensure that the training and test sets adequately represented the minority class, the following strategy was used:

Stratified Train-Test Split and Cross-Validation: Both the training and testing sets accurately reflected the same class distribution proportion as in the original dataset. Model performance was evaluated using 5-fold stratified cross-validation, providing a more robust, unbiased assessment of the model’s generalization.

Feature importance was determined using the permutation importance methodology. This technique assesses the importance of a feature by measuring the decrease in model performance when the feature’s values are randomly shuffled (in the experiments in this study each feature was shuffled 20 times on the test fold). A larger decrease in performance indicates a more important feature. Only those features were included in the feature importance plots that exhibited positive values following summation of the permutation importance score and respective standard deviation across the five test folds. This criterion effectively excluded features with inconsistent or potentially spurious importance.

The age and sex distributions were broadly similar in the national and Tennessee cohorts. The mean age was 48–49 years, with comparable age-group profiles, and men comprised ∼68% of PLWH in both datasets (∼31% women; remainder unknown). Minor differences were present (e.g., a slightly higher proportion of individuals aged 51–60 in Tennessee), but the overall patterns were consistent. In the Tennessee cohort, 3.7% of PLWH had a stroke, while 96.3% did not. The national dataset, while larger, exhibits a more pronounced imbalance, as only 1.0% of PLWH experienced a stroke. The relatively larger percentage of stroke incidence in Tennessee compared with the national cohort may reflect the smaller dataset size in the Tennessee cohort or regional demographic composition (Table 1). These statistics are consistent with the literature, with previous research indicating that between 1% and 5% of patients with HIV experience stroke in clinical case series [3].

Racial composition differed between the cohorts (Table 1): Tennessee had a higher proportion of Black or African American PLWH (62.5% vs. 51.0% nationally) and a lower proportion of White PLWH (31.1% vs. 36.1%); “Other” was a small fraction in both. In both cohorts, stroke cases were disproportionately high among Black or African Americans (national: 59.2% of strokes vs. 50.9% of non-strokes; Tennessee: 76.4% vs. 61.9%), while the White and Other groups accounted for smaller shares of stroke cases (national: 30.9% and 9.9%; Tennessee: 19.6% and 4.0%) (Figures 2A,B), with greater disparities in the Tennessee cohort.

We evaluated XGBoost and random forest using two global performance metrics: AUROC and AUPR (Table 4; Figure 3), estimated from stratified 5-fold cross-validation.

Comparing the feature importance of XGBoost models in predicting an outcome in the National cohort and the Tennessee cohort revealed interesting observations. Hypertension emerged as a consistently important predictor across both groups. Monocyte/white blood cells/leukocytes were important in both models, supporting the role of infection and inflammation in stroke risk [18]. Other potential similarities included age at HIV diagnosis and smoking status, suggesting that these factors play a role in both populations. In general, key factors in both the National and Tennessee cohorts were related to hypertension, smoking, infection, and inflammation, all of which have been associated with increased stroke risk in the past [19]. While there are differences in the relative importance of these factors between the models, both models show that these factors may be important markers of stroke risk in PLWH.

Figure 3A shows the Precision-Recall curve for a Random Forest model evaluated on a national cohort, achieving an AUPR of 0.79. This metric suggests strong performance in identifying and ranking positive cases (i.e., PLWH with stroke) within the dataset. Furthermore, the Random Forest model achieved an AUROC of 0.94 on the National cohort (as depicted in Figure 3B), indicating excellent performance. Interestingly, the performance of the Random Forest model was similar to that of an XGBoost model that achieved an AUPR of 0.81 (Figure 3A) and an AUROC of 0.93 (Figure 3B) on the same National dataset. Despite their different underlying mechanisms, both models leverage the characteristics of the national dataset to achieve comparable levels of performance. The XGBoost model was optimized using early stopping on AUPR and hence had slightly superior performance in that particular metric. Both the Random Forest and XGBoost models demonstrated impressive capabilities for stroke prediction in PLWH using national-level data.

The analysis of feature importance for stroke prediction in PLWH revealed key points between the Random Forest (Figure 4C) and XGBoost (Figure 4A) machine learning models. Both models achieved high predictive accuracy (AUCs of 0.94 and 0.93, respectively) and converged on the importance of certain clinical factors. Both algorithms identified hypertension, albumin levels, white blood cell count, and a history of transient ischemic attack as top predictors of stroke risk in PLWH. Furthermore, several other overlapping features, including hematocrit, age at HIV diagnosis, cholesterol levels, bilirubin levels, smoking status, and various blood chemistry markers, were consistently associated with stroke risk in this specific PLWH population.

The two models also exhibited differences in the specific features and their relative ranking. While both models acknowledged the importance of liver function, the XGBoost model placed greater importance on albumin levels, whereas the Random Forest model assigned more weight to bilirubin (another liver function marker). Also, HDL cholesterol (a lipid marker) was identified as a more influential predictor in the XGBoost model than in the Random Forest model, indicating variations in capturing the relationship between lipid metabolism and stroke risk in PLWH between these algorithms. Variations were also observed for blood chemistry markers, such as carbon dioxide levels and blood urea nitrogen levels. These differences are likely due to the algorithms’ distinct approaches to handling feature interactions.

Figure 5A depicts the weighted average importance score for both models in the National dataset (for calculating, first the importance was scaled by each model’s respective maximum score, then averaged over, and finally rescaled by the mean of the maximum scores of both models). Both models commonly focus on the following clinical features, in decreasing order of their importance: hypertension, white blood cell count, albumin levels, history of transient ischemic attack, monocyte percentage, HDL cholesterol [17, 20], hematocrit, liver function, eosinophil count, age at HIV diagnosis, smoking status, carbon dioxide levels, triglycerides, high cholesterol, basophil percentage, alkaline phosphatase serum, glucose, congestive heart failure, diabetes, and atrial fibrillation in PLWH for stroke prevention, in addition to other markers. Random Forest assigned negligible importance to platelet volume, serum creatinine, serum sodium, hemoglobin, and gender, while XGBoost, apart from hemoglobin, serum sodium, and gender, also designated the following features as unimportant: excessive alcohol usage, blood urea nitrogen, and serum total protein. On average, the models placed less importance on race, creatinine, serum sodium and serum total protein, excessive alcohol usage, gender, and hemoglobin.

The performance of the Random Forest model in identifying strokes within PLWH in Tennessee was similar to that of the XGBoost model. The Random Forest model achieved a mean AUPR of 0.50 (Figure 3A), similar to the performance of XGBoost. This indicates that both models face similar challenges when predicting stroke cases within this specific dataset. Furthermore, the Random Forest model also attained a similar AUROC of 0.82 (Figure 3B) as XGBoost. The consistent AUROC and comparative AUPR values across both models highlight that the characteristics of the Tennessee dataset (including its low availability) are likely the primary drivers of the observed performance, rather than specific model limitations.

When comparing the feature importance for stroke prediction in HIV patients between the XGBoost (Figure 4B) and Random Forest (Figure 4D) models within the Tennessee cohort, the significance of certain key factors was identified. The average importance scores of these models (in the same way as for the national dataset) are plotted in Figure 5B. High cholesterol emerged as a prominent predictor in both models, suggesting its consistent relevance in this specific population. Age at HIV diagnosis, hematocrit, leukocyte count, monocytes in blood, hypertension, glucose lab [20], serum protein, diastolic blood pressure, smoking status, alkaline phosphatase, serum urea nitrogen, serum potassium levels, race [17], diabetes [9, 17, 20], platelet counts, serum calcium [20], body temperature, and congestive heart failure, were other influential shared factors. These overlapping features consistently contributed to stroke prediction across the different algorithmic models within the Tennessee cohort. Features including serum sodium, past transient ischemic attacks, hemoglobin, eosinophil count, atrial fibrillation, excessive alcohol use, basophil counts in blood, gender, and total bilirubin serum levels were at the bottom of the two models, indicating their diminished/negligible contribution.

Performance comparison of the two models on the two cohorts is shown in Table 4. It can be seen from the table that the performance of the two models is comparable despite their different underlying algorithmic architecture. Hence, to isolate the most robust and consistent predictors of stroke in PLWH, a comprehensive comparison of the four feature importance charts (XGBoost and Random Forest for both National and Tennessee cohorts, as depicted in Figure 4) and their weighted average (depicted in Figure 5) was analyzed with a focus on identifying similarities across all models. The following features across both the National and Tennessee cohorts were predictive of stroke in PLWH: hypertension, high cholesterol, white blood cell count/leukocytes, age at HIV diagnosis, serum albumin, monocyte percent, hematocrit, past transient ischemic attack, HDL cholesterol, smoking status, serum glucose, bilirubin, eosinophil count, serum triglycerides, serum alkaline phosphatase, carbon dioxide, serum total protein, blood urea nitrogen, mean platelet volume, congestive heart failure, serum potassium, diabetes, diastolic BP, serum chloride, and race.

The penalized logistic regression was the weakest of the machine learning models in both cohorts, and the size of the shortfall differed sharply between them (Table 4). In the national cohort, it reached an AUPR of 0.23, approximately 5.2 times the random baseline of 4.4%, and an AUROC of 0.79, against compared with AUPR values of 0.81 and 0.79 and AUROC values of 0.93 and 0.94 for XGBoost and random forest, respectively. In the Tennessee cohort, it reached an AUPR of 0.18, approximately 2.3 times the random baseline of 7.8%, and an AUROC of 0.70, against an AUPR of 0.50 and an AUROC of 0.82 for both ensembles. The gap was therefore wider nationally in absolute terms, where the ensembles benefit more from the larger cohort than the linear model does.

The feedforward neural network performed below both ensemble models in each cohort (Table 4). In the national cohort, it reached an AUPR of 0.71 (approximately 16 times the random baseline of 4.4%) and an AUROC of 0.91, against AUPR values of 0.81 and 0.79 and AUROC values of 0.93 and 0.94 for XGBoost and random forest. The shortfall was larger in AUPR than in AUROC, indicating that the network ranked the small number of stroke cases less precisely at the top of the risk distribution while separating the two classes almost as well overall. In the Tennessee cohort, the network achieved an AUPR of 0.40 (approximately 5.1 times the random baseline of 7.8%) and an AUROC of 0.76, against an AUPR of 0.50 and an AUROC of 0.82 for both XGBoost and random forest. Because interval estimates were not computed, these comparisons are descriptive, and we cannot infer a difference between model classes from them. A plausible contributor to the gap is that the tree ensembles route missing values natively during splitting, whereas the network receives mean-imputed inputs in which the pattern of missing laboratory tests is no longer represented. A further contributor in the Tennessee cohort is that the number of training epochs was selected on an internal validation split rather than on the evaluation fold, unlike the ensembles.

The adapted Framingham score performed well below both ensemble models in both cohorts (Table 4). In the Tennessee cohort, it achieved an AUROC of 0.64 and an AUPR of 0.13, approximately 1.7 times the stroke prevalence of 7.8% in the analyzed cohort, indicating modest discrimination and low precision. In the national cohort, it achieved an AUPR of 0.30, approximately 6.8 times the stroke prevalence of 4.4%, but an AUROC of 0.44, below the value of 0.5 expected from random classification. The divergence between the AUROC and AUPR values across the cohorts likely reflects differences in outcome prevalence, population characteristics, and data completeness. The weak performance is attributable to the substitutions required to apply the score, in particular the absence of systolic blood pressure in the national dataset and the use of age at HIV diagnosis in place of current age, and should not be read as a property of the original model. These results indicate that a general population stroke score does not transfer to PLWH EHR cohorts without recalibration and the incorporation of HIV-specific and contemporary clinical risk factors.

Permutation feature importance (Figures 4, 5) was evaluated for the two primary models, random forest and XGBoost; the three comparator models were included for performance comparison only and are reported in Table 4.

In these models, atrial fibrillation, serum sodium, excessive alcohol use, hemoglobin, and gender received low permutation-importance ranks in both cohorts (Figure 5). Apart from these, features such as serum potassium, race, total protein serum, and creatinine were found to be less significant, especially in the national dataset. These factors were found to be more important in the Tennessee cohort, especially total serum protein, as shown in Figure 5B. Platelet count was found to be consistently in the bottom half of the feature importance graphs in both datasets (Figures 5A,B). In the case of the Tennessee cohort, features such as transient ischemic attack, eosinophils, basophils, and bilirubin did not contribute to the model, while they were very important in stroke prediction in PLWH in the National cohort. Previous research has suggested that a history of TIAs is associated with an elevated risk of stroke [36].

Diabetes is an important marker for stroke risk due to its accelerating impact on vascular aging [9, 19, 37]. However, in our analysis, diabetes was found to be in the bottom half of the feature importance figures (Figures 5A,B in both the National and Tennessee cohorts). This may be because diabetes is often associated with abnormalities in laboratory parameters, such as glucose levels, lipid profiles, and inflammatory markers. In our analysis, the ML models attribute more predictive power to these directly measured laboratory values rather than to clinical diabetes diagnoses. For example, elevated serum glucose levels may be a stronger predictor in the model than a diagnosis of diabetes, as shown in Figure 5.

It is well documented that atrial fibrillation and diabetes are strongly and significantly associated with stroke. The discrepancy between the univariable association and the low permutation importance rank arises because correlated features may absorb the signal. Permutation importance measures a feature’s marginal contribution given the other features in the model. Permuting the diabetes variable (DIABETES in Table 2) leaves the serum glucose value intact, and the model recovers the same information from the correlated laboratory value; the same applies to atrial fibrillation relative to the hematologic and cardiovascular features. A low permutation-importance rank under feature correlation is evidence of marginal contribution, not clinical importance.