NephroTox AI's Higher External AUC Comes With 21.7% Coverage

An External Set Tests More Than Internal Fit

Researchers at Macao Polytechnic University developed NephroTox AI to classify drug-associated kidney toxicity from molecular descriptors. The paper compares 28 combinations of seven classification algorithms and four descriptor sets, then evaluates the selected model against compounds held out by source rather than by a random split alone.

The authors initially assembled 2,157 unique compounds from three datasets. Quality checks removed 11 records from the model-construction pool and 70 from the DIRIL source reserved for external evaluation. The final comparison used 1,829 compounds for model development and 247 DIRIL compounds as an external set. Ten different random partitions of the development data supplied internal validation and test results. The external set remained separate.

The source split was important because the three input collections did not assign toxicity labels in the same way. The DIRIL collection drew on FDA labeling and clinical reports, while the other collections used literature or toxicity-database annotations. The researchers standardized molecular records and removed duplicate structures before splitting the data. Conflicting labels were resolved using a stated hierarchy that prioritized FDA information, followed by majority agreement; unresolved ties were excluded. This curation reduced contradictions in the training labels, but it did not turn the combined collections into a single uniform clinical endpoint.

Higher AUC Comes With Much Lower Coverage

Random forest with RDKit two-dimensional descriptors performed best in the benchmark. Its mean AUC was 0.907 in internal validation and 0.896 in the held-out internal test data. On the complete external set, AUC fell to 0.727. These figures answer different questions: a model can rank well within its development data while performing less well on another source's labels and chemical space.

The team used variation among the trees in the random forest to place predictions into four confidence tiers. AUC was 0.902 when analysis was limited to the high- and medium-confidence tiers, but this retained about 54 of the 247 external compounds, or 21.7%. The low- and very-low-confidence tiers contained 88.3% of false negatives and 94.5% of false positives. The filtered AUC therefore describes a smaller selected subset, not improved discrimination over the entire external set.

Confidence thresholds were selected within the training procedure rather than by inspecting the final external outcomes. The resulting confidence classification reflected model uncertainty, separately from a compound’s similarity to the training chemistry. The paper also examined chemical-space overlap: some external errors occurred within well-represented regions, so distance from familiar compounds did not by itself explain every wrong classification.

The Model Flags Uncertainty Without Resolving Mechanism

SHAP analysis ranked partial charge, molecular complexity and molecular weight among influential descriptors. Their contributions explain the model's statistical predictions; they do not establish a kidney-injury pathway. Screening 11,093 DrugBank compounds produced 59 high- or medium-confidence positive predictions, including drugs with known renal toxicity. Network analysis and docking added hypotheses about possible biological connections, not experimental confirmation of newly flagged compounds.

The model uses a binary nephrotoxicity label and two-dimensional molecular information. It does not distinguish tubular, interstitial and glomerular injury, and the authors did not test the newly predicted candidates in a prospective renal-cell assay. A high-confidence flag could help prioritize a compound for a renal assay, while an uncertain prediction could prompt more direct testing rather than be treated as a negative result.