In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective
Researchers have demonstrated that retrieval-augmented generation (RAG) can be understood as an in-context optimization process, where a single linear self-attention layer performs a gradient-descent step on a unified RA…