OFF-CLIP

Improving Normal Detection Confidence in Radiology CLIP
with Simple Off-Diagonal Term Auto-Adjustment

Junhyun Park^1*, Chanyu Moon^2*, Donghwan Lee³, Kyungsu Kim^3†, Minho Hwang^1†

¹Dept. of Robotics and Mechatronics Engineering, DGIST ²Dept. of Electrical Engineering and Computer Science, DGIST ³Seoul National University

* Equal contribution † Corresponding author

MICCAI 2025 · Early accept (top 9%)

Paper arXiv MICCAI Code

Abstract

Zero-shot classification with CLIP-style models is attractive in radiology because it reduces the need for manual labels. However, conventional contrastive training struggles with normal cases, producing many false positives and false negatives. We identify two causes: InfoNCE treats different normal patients as negative pairs, which prevents a cohesive normal cluster, and reports of abnormal patients contain normal sentences that are forced to align with abnormal images.

We propose OFF-CLIP, a simple refinement that introduces an off-diagonal term loss to cluster normal samples, and sentence-level text filtering that removes normal statements from abnormal reports. OFF-CLIP requires no architectural changes. Applied to CARZero, the state-of-the-art zero-shot baseline, it raises normal AUC on VinDr-CXR by 0.61 while maintaining or improving abnormal classification, and it improves pointing-game accuracy for zero-shot grounding.

Why do radiology CLIP models miss normal cases?

Radiology CLIP models such as CARZero are trained with InfoNCE, which treats only the matched image-report pair (the diagonal of the similarity matrix) as positive and every other pair in the batch as negative. In chest X-rays this assumption breaks in two ways.

(a) Conventional diagonal InfoNCE loss, (b) false positives in normal pairs, (c) false negatives in abnormal pairs.

Method

OFF-CLIP pipeline: (1) text prompting, (2) sentence-level labeling, (3) text filtering and pair labeling, followed by the off-diagonal term loss and the abnormal InfoNCE loss.

Problem 1 · False positives in normal pairs

Normal images and normal reports look and read almost the same across patients ("There is no pneumothorax."). InfoNCE still pushes different normal patients apart as negatives (S₁₃, S₃₁ → 0 in (b)), so normal samples never form a cohesive cluster and the model is not confident that an image is normal.

Solution · Off-diagonal term loss

Every entry of the similarity matrix gets a binary cross-entropy target. The diagonal stays positive, and the off-diagonal entries between two normal pairs are also set positive:

\[ \hat{Y}_{ij} = \begin{cases} 1 & \text{if } i = j \ \text{ or both } i, j \text{ are normal}\\ 0 & \text{otherwise} \end{cases} \]

\[ \mathcal{L}_{\mathrm{off}} = -\frac{1}{2B^2}\sum_{i=1}^{B}\sum_{j=1}^{B} \Big[\hat{Y}_{ij}\log\sigma(S_{ij}) + (1-\hat{Y}_{ij})\log\big(1-\sigma(S_{ij})\big) + \hat{Y}_{ji}\log\sigma(S_{ji}) + (1-\hat{Y}_{ji})\log\big(1-\sigma(S_{ji})\big)\Big] \]

This pulls all normal samples together while keeping the standard diagonal alignment.

Problem 2 · False negatives in abnormal pairs

Reports of abnormal patients also contain normal sentences, e.g. "There is no focal consolidation." next to "There are bilateral nodular opacities." Training aligns these normal sentences with an abnormal image (S₂₂ → 1 in (c)) and pushes them away from normal images (S₂₁ → 0), which confuses the boundary between normal and abnormal.

Solution · Text prompting and filtering

Text prompting: GPT-4o extracts the Findings and Impression sections and rewrites them into CARZero-style sentences ("There is pleural effusion at the left lung base.").
Sentence labeling: a pretrained radiology sentence-level anomaly classifier labels each sentence as normal, abnormal or uncertain. A report with at least one abnormal sentence is abnormal; otherwise it is normal. These pseudo-labels need no human annotation.
Filtering: normal and uncertain sentences are removed from abnormal reports, so an abnormal image is aligned only with text that describes its abnormality.

Abnormal InfoNCE and the final objective

To keep fine-grained discrimination between different abnormalities, the standard InfoNCE loss is applied only to the A abnormal pairs in the batch:

\[ \mathcal{L}_{ab} = -\frac{1}{A}\sum_{i=1}^{A}\left[ \log\frac{e^{S^{ab}_{ii}}}{\sum_{j=1}^{A} e^{S^{ab}_{ij}}} + \log\frac{e^{S^{ab}_{ii}}}{\sum_{j=1}^{A} e^{S^{ab}_{ji}}}\right], \qquad \mathcal{L}_{\text{OFF-CLIP}} = \mathcal{L}_{\mathrm{off}} + \lambda_{ab}\,\mathcal{L}_{ab}\ \ (\lambda_{ab}=1) \]

OFF-CLIP changes only the training objective and the text, not the architecture. Following CARZero, it uses a ViT-B/16 image encoder pretrained with M3AE and a BioBERT text encoder, trained on frontal images from MIMIC-CXR.

Results

Zero-shot classification

Applied to CARZero, OFF-CLIP raises normal AUC on all four external test sets, by 0.50 on average (0.61 on VinDr-CXR), and raises total AUC by 0.095 on average, so abnormal classes are maintained or improved rather than traded off.

Dataset	OFF-CLIP	Normal	ATE	CM	EMPH	EFF	NOD	PLT	PNA	Total
VinDr-CXR	✗	0.25	0.74	0.91	0.93	0.94	0.47	0.75	0.90	0.79
VinDr-CXR	✓	0.86	0.84	0.93	0.95	0.94	0.86	0.86	0.83	0.87
Open-I	✗	0.32	0.79	0.89	0.83	0.93	0.55	0.73	0.84	0.72
Open-I	✓	0.74	0.81	0.90	0.91	0.93	0.60	0.76	0.84	0.81
PadChest	✗	0.24	0.78	0.87	0.78	0.95	0.52	0.66	0.80	0.68
PadChest	✓	0.77	0.82	0.88	0.89	0.95	0.64	0.81	0.80	0.76
CheXpert	✗	0.38	0.73	0.86	-	0.92	-	-	0.65	0.73
CheXpert	✓	0.81	0.87	0.88	-	0.92	-	-	0.79	0.86

Zero-shot AUC of CARZero without (✗) and with (✓) OFF-CLIP. ATE: atelectasis, CM: cardiomegaly, EMPH: emphysema, EFF: effusion, NOD: nodule, PLT: pleural thickening, PNA: pneumonia.

Ablation: each component addresses one error type

On Open-I, the off-diagonal term loss alone lifts normal AUC above 0.7, but without filtering the abnormal classes collapse. Text filtering restores abnormal performance, and combining all three components gives the best balance: the gap between FN and FP shares of the errors drops from 0.9986 for the baseline, which labels almost everything abnormal, to 0.01.

Text filter	L_off	L_ab	Normal	ATE	CALC	CM	EMPH	MASS	NOD	OPAC	PLT	Total	\|FN − FP\| share
			0.32	0.79	0.59	0.89	0.83	0.68	0.55	0.79	0.73	0.72	0.9986
	✓		0.72	0.29	0.49	0.34	0.33	0.31	0.47	0.24	0.28	0.32	0.06
	✓	✓	0.75	0.24	0.46	0.30	0.27	0.27	0.44	0.23	0.26	0.29	0.14
✓			0.62	0.58	0.50	0.64	0.46	0.27	0.49	0.31	0.41	0.44	0.46
✓	✓		0.72	0.77	0.55	0.84	0.85	0.76	0.61	0.75	0.74	0.78	0.14
✓	✓	✓	0.74	0.81	0.60	0.90	0.91	0.85	0.60	0.73	0.76	0.81	0.01

Ablation on Open-I with CARZero (AUC). The last column is |FN/(FP+FN) − FP/(FP+FN)|; lower means false positives and false negatives are balanced. CALC: calcification, OPAC: opacity.

Zero-shot grounding

Better separation of normal and abnormal also improves localization. In the pointing game on VinDr-CXR, OFF-CLIP improves calcification from 0.30 to 0.59 with the top 10% attention, and pneumothorax from 0.63 to 1.00 with the top 20%.

Attention	Method	AE	ATE	CALC	CM	CONS	EFF	INF	NOD	OPAC	PLT	PTX
Top 10%	CARZero	0.94	0.61	0.30	0.99	0.84	0.69	0.80	0.47	0.66	0.33	0.63
Top 10%	+ OFF-CLIP	1.00	0.78	0.59	1.00	0.95	0.89	0.90	0.73	0.92	0.47	1.00
Top 20%	CARZero	0.99	0.78	0.59	1.00	0.97	0.79	0.90	0.64	0.83	0.47	0.63
Top 20%	+ OFF-CLIP	1.00	0.89	0.82	1.00	0.97	1.00	0.95	0.84	0.96	0.67	1.00

Pointing-game accuracy on VinDr-CXR using the top 10% / 20% attention per disease prompt. AE: aortic enlargement, CONS: consolidation, INF: infiltration, PTX: pneumothorax.

BibTeX

@inproceedings{park2025off,
  title     = {OFF-CLIP: Improving Normal Detection Confidence in Radiology CLIP with Simple Off-Diagonal Term Auto-Adjustment},
  author    = {Park, Junhyun and Moon, Chanyu and Lee, Donghwan and Kim, Kyungsu and Hwang, Minho},
  booktitle = {Proceedings of the 28th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)},
  year      = {2025}
}