Architectural Deep-Dive: Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

Reinforcement learning with verifiable rewards (RLVR) and Group Relative Policy Optimization (GRPO) have established themselves as standard approaches for post-training advanced reasoning models. While task-level outcome supervision works exceptionally well for formal domains like competitive programming or symbolic math, it exhibits severe structural limitations when applied to multimodal agents performing fine-grained visual reasoning.

When an agent fails to locate a coordinate or misinterprets an engineering blueprint, scalar outcome rewards fail to indicate which specific intermediate reasoning token introduced the error. Integrating on-policy self-distillation (OPSD) using a privileged teacher addresses credit assignment by introducing dense token-level corrections. However, in production pipelines, this introduces an insidious failure mode: privileged-correction contamination.

In standard OPSD setups, a privileged teacher is provided with high-resolution crops, focal patches, or privileged spatial evidence. The student policy is then nudged token-by-token toward the teacher's distribution.

Under empirical scrutiny, the teacher's corrective trajectory often conflates true visual evidence with:

Directly distilling these contaminated token updates poisons the optimization path, entrenching spurious visual shortcuts rather than rigorous perceptual reasoning.

Contaminated Path:   [Visual Input + Text] -> [Privileged Teacher] -> [Linguistic Prior Noise] -> [Corrupted Student Updates]
Attributed Path:     [Visual Input] vs [Degraded View] -> [Attribution Gating] -> [Verified Visual Updates]

To decouple genuine visual corrections from spurious linguistic drift, the Visual-Attribution Gating (VAG) framework enforces a contrastive verification gate prior to accepting any token-level update.

The anchor of the optimization remains the verifier-derived advantage $\hat{A}t$. A corrective candidate token $w_t^$ proposed by the teacher $M*{\text{teacher}}$ is evaluated across two distinct execution contexts:

If the directional alignment of the proposed token update $\Delta \theta_t$ persists identically under $V_{\text{degraded}}$, the token update is classified as non-attributable to the visual evidence and dropped.

import torch
import torch.nn as nn
import torch.nn.functional as F

class VisualAttributionGate(nn.Module):
    def __init__(self, threshold: float = 0.15):
        super().__init__()
        self.threshold = threshold

    def forward(
        self,
        logits_evidence: torch.Tensor,
        logits_degraded: torch.Tensor,
        student_logits: torch.Tensor,
        verifier_advantage: torch.Tensor
    ) -> torch.Tensor:
        """
        Filters teacher distillation updates based on visual attribution.
        """
        p_student = F.softmax(student_logits, dim=-1)
        p_evidence = F.softmax(logits_evidence, dim=-1)
        p_degraded = F.softmax(logits_degraded, dim=-1)

        kl_evidence = F.kl_div(p_student.log(), p_evidence, reduction='none').sum(dim=-1)
        kl_degraded = F.kl_div(p_student.log(), p_degraded, reduction='none').sum(dim=-1)

        attribution_delta = kl_evidence - kl_degraded

        gate_mask = (attribution_delta > self.threshold) & (verifier_advantage > 0)

        return gate_mask.float()

Canonical publication: https://bohdanhordii.dev

Author: Bohdan Hordii | Website: bohdanhordii.dev | Contact: @bohdan_hordii