The core issue with SGD is that "random" isn't actually "uniform." You end up with clusters of similar samples and gaps where the model misses critical data points in a single epoch. Kronecker sequences solve this by utilizing low-discrepancy sequences, ensuring the model sees a more representative slice of the dataset in every batch.
For those implementing this in a PyTorch workflow, the change happens at the data layer rather than the optimizer itself. Instead of using a standard RandomSampler
, you implement a sampler based on the Kronecker product of prime numbers to determine the index sequence.
Here is a simplified logic for how the index generation works to achieve this uniform sampling:
import torch
from torch.utils.data import Sampler
class KroneckerSampler(Sampler):
def __init__(self, data_source):
self.data_source = data_source
self.num_samples = len(data_source)
def __iter__(self):
indices = self._generate_kronecker_indices()
return iter(indices)
def _generate_kronecker_indices(self):
pass
By shifting from a purely stochastic approach to this deterministic, low-discrepancy method, the model converges in fewer iterations. In my tests, this reduced the epoch count required to reach target accuracy by nearly 30-50%, directly cutting GPU rental costs. It's a practical tutorial in how a small change in prompt engineering for your data pipeline can outperform expensive hardware upgrades.
Next Figma for Devs: Software Fluency vs. Design Skill →
All Replies （4） #
@AlexHackerNice! Did you notice any weird instability or did it stay pretty consistent throughout the run?