Event-Frame Fusion for Inter-Frame Segmentation via Event-Guided Motion

A hybrid vision architecture combining frame-based and event-based cameras achieves segmentation rates up to 500 Hz at under 1.87 mJ per inference, according to an arXiv paper (arXiv:2609.22500v1). The system pairs a 42k-parameter spiking neural network for motion estimation with a 0.84M-parameter event-driven SNN for frame-based semantic segmentation that interpolates motion between frames, while maintaining real-time GPU execution up to 200 Hz. The authors report the approach compensates for information loss in frames affected by blur or overexposure, addressing the 20-30 FPS temporal resolution limit of conventional cameras in autonomous navigation.

arXiv:2609.22500v1 Announce Type: new Abstract: Autonomous navigation requires precise and efficient semantic segmentation, yet existing frame-based approaches remain limited by motion blur, glare, latency, and the low temporal resolution 20-30 FPS of conventional cameras, which leads to information loss between frames. Event cameras have emerged as an alternative sensing modality, capturing intensity changes asynchronously with high temporal resolution, high dynamic range, and sparse outputs. However, event-based algorithms still fall short of frame-based ones in accuracy, as most segmentation methods are designed for dense frame data. To overcome these limitations, we propose a hybrid vision architecture that combines conventional frame-based and event-based cameras. The system integrates two complementary components: 1 a compact Spiking Neural Network SNN with 42k parameters for motion estimation, and 2 a lightweight event-driven SNN with 0.84M parameters for frame-based semantic segmentation, which interpolates motion between frames to refine segmentation results. By predicting inter-frame segmentations, the framework achieves segmentation rates of up to 500 Hz with an energy consumption below 1.87 mJ per inference, while maintaining real-time GPU execution at frequencies up to 200 Hz. Additionally, our approach compensates for information loss in frames affected by blur or overexposure, enabling more robust perception in challenging conditions.