arXiv:2611.23591v1 Announce Type: new Abstract: Unlabeled test data can improve the zero-shot classifier of a vision-language model, yet transductive and test-time adaptation methods often fall below the zero-shot baseline when only a few vocabulary classes are present in a batch or when streams are correlated. We trace this failure to a shared design: every class owns independent parameters estimated from its own samples, so absent or rare classes are left at biased text anchors or fitted to foreign samples. We propose Graph-Regularized Anchor-displacement Field Transduction (GRAFT), which models the adapted prototypes as text anchors plus a displacement field that is partially pooled over a semantic class graph through a Gaussian Markov random field prior, together with an explicit class-presence model. Inference is closed-form and gradient-free on cached features. Anchor-only shrinkage and a single global shift are the two extreme settings of the prior; absent classes are provably harmonic extensions of the present ones, and the pooling strength can be chosen by empirical Bayes. Under the realistic batch and online protocols on eleven benchmarks and four ImageNet shifts, GRAFT improves over the zero-shot model in every setting and over anchor-based transduction where few classes are present.
GRAFT: Graph-Regularized Prototype Fields for Realistic Test-Time Adaptation of VLMs
Researchers proposed GRAFT (Graph-Regularized Anchor-displacement Field Transduction), a gradient-free test-time adaptation method that models adapted prototypes as text anchors plus a displacement field partially pooled over a semantic class graph via a Gaussian Markov random field prior, according to arXiv paper 2611.23591v1. Across eleven benchmarks and four ImageNet shifts, GRAFT improved over the zero-shot vision-language model in every realistic batch and online setting and beat anchor-based transduction where few classes are present. The authors trace prior adaptation failures to per-class independent parameters that leave absent or rare classes at biased text anchors or fitted to foreign samples.
When adapting a VLM on unlabeled test batches with few classes, replace independent per-class prototypes with graph-pooled anchor displacements and an explicit class-presence model; GRAFT beat zero-shot in every tested setting.
more in playbook: Evals and testing for LLM appsRun your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.