Population-Level Medical AI Using 3D Self-Supervised Learning

Our self‑supervised Vision Transformer learned from 300 000 CT, MRI, and mammogram organ images without labels, achieving 91 % k‑Nearest Neighbor accuracy across 130 categories.

Our self‑supervised model learned how these ~1 000 brains relate to one another — without any labels — and organized them into a small‑world, scale‑free network: About 5 % of brains act as “hub‑brains” (prototypes) at the center. The rest branch outward like relations in a social network, only a few “steps” away from any other brain (think six degrees of separation).

Notice the split: the large cluster on the left is entirely male, while the large cluster on the right mixes male and female brains. We don’t yet know why — but this is the power of discovery: the model finds patterns we didn’t expect, giving us new questions to explore.

Self‑supervised learning organizes anatomy into scale‑free, small‑world graphs — the same kind of networks we see in biology and even social systems. In simple terms, the model is finding the most efficient way to connect similar organs, like neighborhoods linked by a few well‑connected “hubs.”

This isn’t something we programmed in — it happens on its own, like a phase change when water freezes. And that’s where the discoveries begin. For example, some organs, like the bladder, cluster by biological sex, while many others, like brains (see above), do not. This is tremendously interesting!

Another surprise: these learned representations don’t follow the familiar “bell curve” (Gaussian distribution) that so much of medical science assumes. Instead, they have a long tail — meaning rare, unusual cases carry far more weight than we typically expect.

To expand on how remarkable this is, there is one patient in our dataset that has 4 torso CT scans. We use one CT scan, select 45 different organs including L-vertebrae, pancreas, gallbladder, duodenum, stomach, left and right adrenal glands, heart ventricles, kidneys, psoas, glutes, and more - a good variety of organs. Since this patient has 4 scans, the search scan is held out meaning that there are 3 instances of each of his organs in out 300k organ dataset. Our search finds his other organs instances, with all 3 other instances of each organ as the top 3 search results, 65% of the time, and this is with a tiny research-sized model too. To reiterate just how sensitive the search is, out model finds this patient's L1-L5 vertebrae as the top 3 search results 100% of the time. So, inputing his L3 vertebra as a search query, the top 3 search results are his other three L3 vertebrae - out dataset has ~6000 L3 vertebrae and 52,000 vertebra of all types in it. The model doesn't just organize by organ type, but it clusters anatomy at the level of the individual.

In each video, multiple panes are shown from left to right. The first pane displays the original scan image, while each successive pane shows three principal components: the first component mapped to the red channel, the second to green, and the third to blue. Moving across the panes from left to right, the principal components increase in groups of three (e.g., components 1–3, then 4–6, then 7–9, and so on).

The videos include a BIRADS 5 tomosynthesis (3D) mammogram, a T1‑weighted contrast‑enhanced (T1Gd) MRI of the brain, a CT of a left kidney, and a CT of a liver.

It's especially obvious in the liver video that the model learns to focus on the correct anatomy despite having no labels or annotations during training — this focus emerges automatically.

The U-Net architecture was a game‑changer for medical image segmentation. It uses a clever design with an encoder–decoder setup connected by “skip connections,” which pass information directly from the early layers of the network to the later ones. This gives the model strong “hints” about what the correct segmentation should look like. But these shortcuts come with downsides: they can cause the model to rely too heavily on those direct connections, making it less flexible when dealing with unfamiliar data. They can also introduce small visual errors—often called “floaters”—in the final results.

To overcome these issues, we built a new segmentation model that removes skip connections entirely. This forces the network to build a deeper understanding of the images it sees, improving its ability to handle new, unseen data and reducing visual artifacts in its outputs.

Our model also uses a Vision Transformer—a type of neural network that can “look” at an image as a whole, not just piece by piece. This gives it a global perspective and opens the door for future features like prompt‑based segmentation. Even without prompting, our model achieves a 94% DICE score on the TotalSegmentator dataset, performing on par with leading models like nnUnet but with greater consistency.

The system is built for scale, supporting distributed training and inference using PyTorch. We’ll be open‑sourcing the model soon so others can explore and build on it.