Architectural Deep-Dive: Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

Bohdan Hordii published an architectural deep-dive on Visual-Attribution Gating (VAG), a framework that filters on-policy self-distillation updates in multimodal reinforcement learning to prevent what he calls 'privileged-correction contamination.' The approach runs a contrastive verification gate that compares a teacher's proposed token update against a degraded visual view, dropping updates whose directional alignment persists without genuine visual evidence. Hordii reports that directly distilling contaminated teacher trajectories entrenches spurious visual shortcuts rather than rigorous perceptual reasoning.

Reinforcement learning with verifiable rewards RLVR and Group Relative Policy Optimization GRPO have established themselves as standard approaches for post-training advanced reasoning models. While task-level outcome supervision works exceptionally well for formal domains like competitive programming or symbolic math, it exhibits severe structural limitations when applied to multimodal agents performing fine-grained visual reasoning. When an agent fails to locate a coordinate or misinterprets an engineering blueprint, scalar outcome rewards fail to indicate which specific intermediate reasoning token introduced the error. Integrating on-policy self-distillation OPSD using a privileged teacher addresses credit assignment by introducing dense token-level corrections. However, in production pipelines, this introduces an insidious failure mode: privileged-correction contamination . In standard OPSD setups, a privileged teacher is provided with high-resolution crops, focal patches, or privileged spatial evidence. The student policy is then nudged token-by-token toward the teacher's distribution. Under empirical scrutiny, the teacher's corrective trajectory often conflates true visual evidence with: Directly distilling these contaminated token updates poisons the optimization path, entrenching spurious visual shortcuts rather than rigorous perceptual reasoning. php Contaminated Path: Visual Input + Text - Privileged Teacher - Linguistic Prior Noise - Corrupted Student Updates Attributed Path: Visual Input vs Degraded View - Attribution Gating - Verified Visual Updates To decouple genuine visual corrections from spurious linguistic drift, the Visual-Attribution Gating VAG framework enforces a contrastive verification gate prior to accepting any token-level update. The anchor of the optimization remains the verifier-derived advantage $\hat{A} t$. A corrective candidate token $w t^ $ proposed by the teacher $M {\text{teacher}}$ is evaluated across two distinct execution contexts: If the directional alignment of the proposed token update $\Delta \theta t$ persists identically under $V {\text{degraded}}$, the token update is classified as non-attributable to the visual evidence and dropped. python import torch import torch.nn as nn import torch.nn.functional as F class VisualAttributionGate nn.Module : def init self, threshold: float = 0.15 : super . init self.threshold = threshold def forward self, logits evidence: torch.Tensor, logits degraded: torch.Tensor, student logits: torch.Tensor, verifier advantage: torch.Tensor - torch.Tensor: """ Filters teacher distillation updates based on visual attribution. """ Compute KL divergence shifts from student p student = F.softmax student logits, dim=-1 p evidence = F.softmax logits evidence, dim=-1 p degraded = F.softmax logits degraded, dim=-1 Directional impact of visual evidence kl evidence = F.kl div p student.log , p evidence, reduction='none' .sum dim=-1 kl degraded = F.kl div p student.log , p degraded, reduction='none' .sum dim=-1 Visual Attribution Score: Difference in distribution divergence attribution delta = kl evidence - kl degraded Gate condition: update must be positively grounded in visual evidence and aligned with task-level verifier advantage gate mask = attribution delta self.threshold & verifier advantage 0 return gate mask.float Canonical publication: https://bohdanhordii.dev https://bohdanhordii.dev Author: Bohdan Hordii | Website: bohdanhordii.dev | Contact: @bohdan hordii