OFF-CLIP | Chanyu Moon

Period: 2024.06 – 2025.09 · Role: Co-first author. Identified the first problem, formulated the off-diagonal term loss, and implemented all baseline models for benchmarking.

Problem

Vision-language models that detect abnormalities in chest X-rays suffer from high false-positive rates on normal images. Analyzing the data, we found two bottlenecks:

Normal images and reports are semantically similar across patients, yet conventional InfoNCE treats different normal patients as negative pairs, which breaks up the normal cluster.
Reports of abnormal patients also contain normal sentences (e.g. “There is no pneumothorax.”). Forcing these to align with abnormal images degrades performance.

(a) Conventional diagonal InfoNCE loss, (b) false positives in normal pairs, (c) false negatives in abnormal pairs.

Method

Off-diagonal term loss: treat all normal image-text pairs as positives.
Text filtering: an LLM rewrites each report into sentences, a sentence-level anomaly classifier labels them, and normal sentences are removed from abnormal reports.

OFF-CLIP pipeline: text prompting and filtering, off-diagonal term loss, and abnormal InfoNCE loss.

Results

Applied to CARZero, OFF-CLIP raises zero-shot normal AUC from 0.25 → 0.86 on VinDr-CXR, 0.32 → 0.74 on Open-I, 0.24 → 0.77 on PadChest, and 0.38 → 0.81 on CheXpert, while maintaining or improving abnormal-class performance.

Publication: (Park* et al., 2025)

References

2025

MICCAI
OFF-CLIP: Improving Normal Detection Confidence in Radiology CLIP with Simple Off-Diagonal Term Auto-Adjustment

Junhyun Park^*, Chanyu Moon^*, Donghwan Lee, and 2 more authors

In Proceedings of the 28th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Early accept (top 9%) , 2025

Abs arXiv Bib HTML PDF

Contrastive Language-Image Pre-training (CLIP) models for radiology suffer from high false-positive rates on normal images. We identify two causes: conventional InfoNCE treats different normal patients as negative pairs, preventing a cohesive normal cluster, and reports of abnormal patients contain normal sentences that are forced to align with abnormal images. OFF-CLIP addresses these with an off-diagonal term loss that treats all normal samples as positive pairs, and with sentence-level text filtering that removes normal descriptions from abnormal reports. Applied to CARZero, OFF-CLIP substantially improves normal detection while maintaining abnormal classification performance on VinDr-CXR, Open-I, PadChest, and CheXpert.
@inproceedings{park2025off, title = {OFF-CLIP: Improving Normal Detection Confidence in Radiology CLIP with Simple Off-Diagonal Term Auto-Adjustment}, author = {Park, Junhyun and Moon, Chanyu and Lee, Donghwan and Kim, Kyungsu and Hwang, Minho}, booktitle = {Proceedings of the 28th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)}, year = {2025}, }