Learning Segmentation from Radiology Reports

Tumor segmentation in CT scans is key for diagnosis, surgery, and prognosis, yet segmentation masks are scarce because their creation requires time and expertise. Public abdominal CT datasets have from dozens to a couple thousand tumor masks, but hospitals have hundreds of thousands of tumor CTs with radiology reports. Thus, leveraging reports to improve segmentation is key for scaling. In this paper, we propose a report-supervision loss ( Chemical structure diagram showing a complex organic molecule with a central six-membered carbon ring. Attached to the ring are various functional groups, including a hydroxyl group (OH) and a carboxyl group (COOH). The structure includes multiple double bonds and a side chain extending from the ring. The diagram is labeled with standard chemical notation.) that converts radiology reports into voxel-wise supervision for tumor segmentation AI. We created a dataset with 6,718 CT-Report pairs (from the UCSF Hospital), and merged it with public CT-Mask datasets (from AbdomenAtlas 2.0). We used our Flow chart illustrating a process with interconnected nodes and arrows. Central node labeled "Start" leads to multiple decision points and actions, including "Process A," "Decision B," and "End." Arrows indicate the flow direction, showing pathways and outcomes based on decisions. The chart visually represents a sequence of steps and choices in a structured format. to train with these masks and reports, and strongly improved tumor segmentation in internal and external validation—F1 Score increased by up to 16% with respect to training with masks only. By leveraging readily available radiology reports to supplement scarce segmentation masks, Flow chart illustrating a process with interconnected nodes and arrows. The chart begins with a central node labeled "Start" and branches into multiple paths, each leading to different outcomes. Key terms include "Decision," "Process," and "End." The flow chart visually represents decision-making steps and pathways, emphasizing logical progression and outcomes. strongly improves AI performance both when very few training masks are available (e.g., 50), and when many masks were available (e.g., 1.7K). Project: https://github.com/MrGiovanni/R-Super