Modular Parameter-Efficient Continual Learning in Vision Transformers: From Domain Shifts to Class Streams

Visual recognition systems deployed over time are increasingly expected to adapt to new domains and new categories while preserving prior knowledge, limiting reliance on raw historical images, and keeping inference cost bounded. Pre-trained Vision Transformers provide a strong foundation because their frozen representations transfer across many visual tasks, but a frozen backbone does not by itself solve continual learning. The remaining question is where to place the small amount of plasticity that is still needed, how to preserve it, and how to route it when task or domain identity is unavailable. This dissertation answers that question with a modular view of parameter-efficient continual learning: adaptation should be organized as small, role-matched corrections rather than as full-model retraining or whole-expert selection. The dissertation develops this view across two continual-learning settings. In domain incremental learning, the label space is shared but the visual distribution changes over time. The first domain-track method, prompt-based domain incremental learning, adds per-domain prompts and modular classifier heads on top of a frozen Vision Transformer. This design captures domain-specific context and decision boundaries without duplicating full backbones or rehearsing old images. The second domain-track method, preference-guided dual-space residual composition, addresses the hidden-domain routing problem more directly. Instead of selecting the expert whose source domain looks most similar to the input, it learns to compose feature-space adapter residuals and logit-space residual heads according to which corrections improve classification. This shifts routing from domain recovery to prediction support. In class incremental learning, new categories arrive sequentially and the model must classify over all seen classes. Hybrid Prototype-Statistical Replay studies the shared-adapter setting, where inference cost remains fixed but a single adapter is vulnerable to drift. It stabilizes different modules with different image-free summaries: class prototypes preserve classifier geometry, while Gaussian state statistics preserve intermediate adapter behavior. LVP-Route studies the complementary setting of task-specific adapter banks. It keeps adapter specialization but uses semantic prototype pools to select candidate tasks, softly mix adapter outputs, and cap the number of adapted forwards at inference. Across the four works, the central finding is that continual adaptation is most effective when each lightweight component is trained, replayed, and routed according to the function it serves. Domain-specific prompts and heads capture visual shifts without retaining old data; preference-guided residual routing improves hidden-domain prediction by optimizing expert usefulness rather than domain identity; module-specific replay prevents shared-adapter drift without increasing inference cost; and semantic soft routing preserves task-specific specialization while controlling computation. Together, these results show that the useful unit of continual adaptation is not the entire model, but the smallest correction that preserves, redirects, or extends a frozen visual representation.