ZeroR@CHiPSAL 2026: Two-Stage Vision-Language Adaptation with Contrastive Learning for Nepali Meme Classification

A team from ZeroR achieved 2nd place on hate speech detection (F1: 0.797) and 4th place on sentiment analysis (F1: 0.518) in the CHiPSAL 2026 shared task on multimodal hate speech and sentiment detection in Nepali memes, using a two-stage vision-language adaptation of the Qwen3-VL-8B-Instruct model with contrastive learning. The approach, based on the RA-HMD framework, combines LoRA fine-tuning with an MLP projection head and contrastive backbone fine-tuning with supervised InfoNCE loss, handling class imbalance via oversampling, augmentation, and focal loss. The system leverages native Devanagari understanding to avoid error propagation from separate OCR and translation pipelines.

arXiv:2607.28637v1 Announce Type: new Abstract: This paper presents our system for the CHiPSAL 2026 shared task on multimodal hate speech and sentiment detection in Nepali memes. We address both subtasks: binary hate speech classification and three-class sentiment analysis. Our approach adapts the Robust Adaptation of Hateful Meme Detection RA-HMD framework using Qwen3-VL-8B-Instruct, a state-of-the-art vision-language model with native Devanagari support. We employ a two-stage training pipeline: 1 LoRA fine-tuning with an MLP projection head for generative classification, and 2 contrastive backbone fine-tuning with supervised InfoNCE loss. We handle class imbalance through minority oversampling, image augmentation, and focal loss. At inference, we ensemble Stage 1 token probabilities with Stage 2 classifier scores using validation-tuned weights. Our end-to-end approach eliminates error propagation from separate OCR and translation pipelines by leveraging the model's native Devanagari understanding. Our system achieved \textbf{2nd place} on hate speech detection F1: 0.797 and \textbf{4th place} on sentiment analysis F1: 0.518 . We provide detailed ablations, error analysis, and insights into adapting large vision-language models for low-resource South Asian languages.