Efficient Contrastive Learning

Period: 2025.02 – 2025.10 · Role: Co-first author. Led the mathematical formulation of the first-order Taylor approximation.

Problem

The standard InfoNCE loss requires a batch-by-batch similarity matrix, so memory and compute grow quadratically with batch size.

Method

Writing the second term of InfoNCE as \(f(t) = \log \sum_j \exp(st)\) and expanding around \(t = 0\) gives \(f(t) \simeq \log B + \left(\frac{1}{B}\sum_j s\right) t\). Substituting \(s = r_i \cdot z_j\) and \(t = 1\):

\[\mathcal{L}_{ap} = -\frac{1}{B} \sum_{i}^{B} (r_i \cdot z_i) + \log B + \left( \frac{1}{B} \sum_{i}^{B} r_i \right) \cdot \left( \frac{1}{B} \sum_{j}^{B} z_j \right)\]

The cross term reduces to a dot product of batch means, so no similarity matrix is ever formed.

Results

Integrated into TS2Vec and evaluated on UCI HAR:

Method	Accuracy	AUPRC	Time (s)	GPU peak (MB)	Throughput (items/s)
Original	0.9213	0.9485	60.1	20576.0	122.4
Taylor	0.9087	0.9291	31.9	8214.0	230.6

Time, peak memory, speedup (up to 7.3x), and memory reduction (up to 8.5x) across model configurations.

Publication: (Jeong* et al., 2025)

References

2025

ICTC
Efficient Biomedical Time-series Contrastive Learning: Revisiting Taylor Expansion in Loss Optimization

Yunho Jeong^*, Chanyu Moon^*, Jinmo Kim, and 1 more author

In 2025 16th International Conference on Information and Communication Technology Convergence (ICTC), 2025

Abs Bib HTML

Contrastive learning has become a cornerstone for self-supervised time-series representation learning, yet its practical application is often hindered by the quadratic computational and memory complexity of the standard InfoNCE loss. We leverage a first-order Taylor expansion to approximate the InfoNCE loss, circumventing the explicit computation of the full similarity matrix and reducing the complexity to a linear scale. Integrated into TS2Vec and evaluated on the UCI Human Activity Recognition benchmark, our approach achieves up to a 7.3x speedup and an 8.5x reduction in peak memory usage while maintaining classification accuracy comparable to the original model.
@inproceedings{jeong2025efficient, title = {Efficient Biomedical Time-series Contrastive Learning: Revisiting Taylor Expansion in Loss Optimization}, author = {Jeong, Yunho and Moon, Chanyu and Kim, Jinmo and Choi, Ji-Woong}, booktitle = {2025 16th International Conference on Information and Communication Technology Convergence (ICTC)}, pages = {146--151}, year = {2025}, organization = {IEEE}, }