* Corresponding Author: rs2492@cornell.edu
* Corresponding Author: rs2492@cornell.edu
BackSplit improves lesion segmentation by sub-dividing the background into semantically meaningful auxiliary classes. Conventional lesion segmentation collapses all non-lesion regions into a single background, discarding the anatomical context and often producing false positives. BackSplit refines the background into semantically meaningful auxiliary structures (e.g., organ parenchyma) that are learned jointly with the lesion target. This structured background supervision enriches contextual understanding, yielding sharper lesion boundaries, fewer false detections, and theoretically more stable predictions — consistent with higher expected Fisher Information and reduced estimator variance compared to conventional binary training.
Segmenting small lesions in medical images remains notoriously difficult. Most prior work tackles this challenge by either designing better architectures, loss functions, or data augmentation schemes; and collecting more labeled data. We take a different view, arguing that part of the problem lies in how the background is modeled. Common lesion segmentation collapses all non-lesion pixels into a single "background" class, ignoring the rich anatomical context in which lesions appear. In reality, the background is highly heterogeneous — composed of tissues, organs, and other structures that can now be labeled manually or inferred automatically using existing segmentation models.
In this paper, we argue that training with fine-grained labels that sub-divide the background class, which we call BackSplit, is a simple yet powerful paradigm that can offer a significant performance boost without increasing inference costs. From an information theoretic standpoint, we prove that BackSplit increases the expected Fisher Information relative to conventional binary training, leading to tighter asymptotic bounds and more stable optimization.
With extensive experiments across multiple datasets and architectures, we empirically show that BackSplit consistently boosts small-lesion segmentation performance, even when auxiliary labels are generated automatically using pretrained segmentation models. Additionally, we demonstrate that auxiliary labels derived from interactive segmentation frameworks exhibit the same beneficial effect, demonstrating its robustness, simplicity, and broad applicability.
BackSplit decomposes the background into multiple auxiliary classes and jointly optimizes them alongside the lesion target, thereby improving the supervision signal for the primary segmentation objective. During inference, the model predicts the target class while implicitly leveraging the contextual knowledge learned from the auxiliary background classes, even though they are not explicitly used.
We formally demonstrate that multi-class training yields higher expected Fisher Information than coarsened binary training, therefore providing more statistically efficient predictions for the target class. In binary training, the background gradient conflates signals from multiple anatomical structures. Decomposing it into semantically distinct support structures yields more disentangled and informative gradients, leading to sharper curvature and higher Fisher Information.
BackSplit is architecture-agnostic and consistently improves performance. We evaluate BackSplit across three widely adopted segmentation backbones (U-Net, ResEncU-Net, and SegResNet) on five diverse datasets spanning CT, MRI, and PET modalities.
Across all datasets and architectures, BackSplit consistently improves performance (measured with Dice, normalized surface distance, and the 95th-percentile Hausdorff distance) without increasing model parameters. The auxiliary classes typically correspond to organs or tissues adjacent to the target structure, demonstrating that structured background supervision improves lesion delineation without modifying the network architecture.
BackSplit works with automatically derived auxiliary segmentations from large pretrained models such as TotalSegmentator and VIBE-Segmentator, as well as noisy labels from interactive segmentation frameworks. Even highly noisy, interactively generated auxiliary cues can enhance target segmentation performance under the BackSplit paradigm, highlighting its robustness to imperfect supervision.
If you find our work or any of our materials useful, please cite our paper: