# Architectural Deep-Dive: Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

> Source: <https://dev.to/hordii_bohdan/architectural-deep-dive-reinforcement-learning-with-visually-attributable-on-policy-430p>
> Published: 2026-10-20 07:10:41+00:00

Reinforcement learning with verifiable rewards (RLVR) and Group Relative Policy Optimization (GRPO) have established themselves as standard approaches for post-training advanced reasoning models. While task-level outcome supervision works exceptionally well for formal domains like competitive programming or symbolic math, it exhibits severe structural limitations when applied to multimodal agents performing fine-grained visual reasoning.

When an agent fails to locate a coordinate or misinterprets an engineering blueprint, scalar outcome rewards fail to indicate which specific intermediate reasoning token introduced the error. Integrating on-policy self-distillation (OPSD) using a privileged teacher addresses credit assignment by introducing dense token-level corrections. However, in production pipelines, this introduces an insidious failure mode: **privileged-correction contamination**.

In standard OPSD setups, a privileged teacher is provided with high-resolution crops, focal patches, or privileged spatial evidence. The student policy is then nudged token-by-token toward the teacher's distribution.

Under empirical scrutiny, the teacher's corrective trajectory often conflates true visual evidence with:

Directly distilling these contaminated token updates poisons the optimization path, entrenching spurious visual shortcuts rather than rigorous perceptual reasoning.

``` php
Contaminated Path:   [Visual Input + Text] -> [Privileged Teacher] -> [Linguistic Prior Noise] -> [Corrupted Student Updates]
Attributed Path:     [Visual Input] vs [Degraded View] -> [Attribution Gating] -> [Verified Visual Updates]
```

To decouple genuine visual corrections from spurious linguistic drift, the Visual-Attribution Gating (VAG) framework enforces a contrastive verification gate prior to accepting any token-level update.

The anchor of the optimization remains the verifier-derived advantage $\hat{A}*t$. A corrective candidate token $w_t^*$ proposed by the teacher $M*{\text{teacher}}$ is evaluated across two distinct execution contexts:

If the directional alignment of the proposed token update $\Delta \theta_t$ persists identically under $V_{\text{degraded}}$, the token update is classified as non-attributable to the visual evidence and dropped.

``` python
import torch
import torch.nn as nn
import torch.nn.functional as F

class VisualAttributionGate(nn.Module):
    def __init__(self, threshold: float = 0.15):
        super().__init__()
        self.threshold = threshold

    def forward(
        self,
        logits_evidence: torch.Tensor,
        logits_degraded: torch.Tensor,
        student_logits: torch.Tensor,
        verifier_advantage: torch.Tensor
    ) -> torch.Tensor:
        """
        Filters teacher distillation updates based on visual attribution.
        """
        # Compute KL divergence shifts from student
        p_student = F.softmax(student_logits, dim=-1)
        p_evidence = F.softmax(logits_evidence, dim=-1)
        p_degraded = F.softmax(logits_degraded, dim=-1)

        # Directional impact of visual evidence
        kl_evidence = F.kl_div(p_student.log(), p_evidence, reduction='none').sum(dim=-1)
        kl_degraded = F.kl_div(p_student.log(), p_degraded, reduction='none').sum(dim=-1)

        # Visual Attribution Score: Difference in distribution divergence
        attribution_delta = kl_evidence - kl_degraded

        # Gate condition: update must be positively grounded in visual evidence
        # and aligned with task-level verifier advantage
        gate_mask = (attribution_delta > self.threshold) & (verifier_advantage > 0)

        return gate_mask.float()
```

Canonical publication: [https://bohdanhordii.dev](https://bohdanhordii.dev)

Author: Bohdan Hordii | Website: bohdanhordii.dev | Contact: @bohdan_hordii