cd /news/machine-learning/provable-cross-modal-transfer-of-hie… · home › topics › machine-learning › article
[ARTICLE · art-170303] src=arxiv.org ↗ pub= topic=machine-learning verified=true sentiment=· neutral

Provable Cross-Modal Transfer of Hierarchical Concept Structure in CLIP Embeddings

A new arXiv paper (2611.17671v1) formalizes Hierarchical Angular Separation (HAS), a geometric property in which each non-leaf concept embedding in CLIP is more similar to its descendants than to off-branch concepts, and proves that this hierarchy transfers from images to text. The authors show HAS holds approximately in raw images and randomly initialized CLIP image embeddings, but in text embeddings only after contrastive learning, and analyze an unconstrained features model in which image embeddings are fixed and only text embeddings are trained. At the global minimizer, the text-side HAS violation percentage at each node is at most the image-side HAS violation and near-violation percentage, a result the authors attribute to the hierarchical co-occurrence structure of image-text pairs and extend to the InfoNCE loss under sufficient regularization, with synthetic experiments validating the theory.

by read1 min views3 publishedNov 16, 2026

arXiv:2611.17671v1 Announce Type: new Abstract: Contrastive vision-language models such as CLIP are trained to align images and text, not to represent taxonomies. Yet empirical studies show that CLIP embeddings exhibit hierarchical structure aligned with the hierarchy of semantic concepts. A theoretical characterization of why such hierarchy emerges from a non-hierarchical objective remains open. In this paper, we formalize this phenomenon as Hierarchical Angular Separation (HAS), a geometric property whereby each non-leaf concept embedding is more similar to its descendants than to off-branch concepts. Empirically, we show that HAS holds approximately in raw images and randomly initialized CLIP image embeddings, while in text embeddings it holds approximately only after contrastive learning. To theoretically explain this asymmetry, we analyze an unconstrained features model in which the image embeddings are fixed and only the text embeddings are trained. At the global minimizer, the text-side HAS violation percentage at each node is at most the image-side HAS violation and near-violation percentage. The key mechanism is the hierarchical co-occurrence structure of image-text pairs, which biases text embeddings toward root-centered image prototypes. For a linear contrastive loss, we derive a closed-form solution that makes this mechanism explicit. We extend the analysis to the InfoNCE loss and establish analogous guarantees under sufficient regularization. Synthetic experiments validate the theory and show that HAS is transferred even beyond the regimes covered by our analysis.

── more in #machine-learning 4 stories · sorted by recency
── more on @clip 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/provable-cross-modal…] indexed:0 read:1min 2026-11-16 · —