Build a RAG-Based AI Assistant in Kotlin with a Vector Database

A normal LLM answers questions from information contained in its model. A Retrieval-Augmented Generation (RAG) system adds an external knowledge layer so an application can answer questions using private or frequently changing documents.

In this tutorial, we will build the architecture for a Kotlin client that communicates with a backend RAG service.

Suppose an application contains company documentation.

Instead of sending the entire documentation to an LLM for every question, the system can:

Documents
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector Database

At query time:

User Question
   ↓
Embedding
   ↓
Vector Search
   ↓
Relevant Chunks
   ↓
LLM
   ↓
Answer

A clean Android architecture can look like:

Compose UI
   |
ViewModel
   |
RagRepository
   |
API Client
   |
RAG Backend

The vector database and LLM should normally remain on the backend rather than being exposed directly to the mobile application.

Define a request:

data class AskRequest(
    val question: String,
    val conversationId: String?
)

And a response:

data class AskResponse(
    val answer: String,
    val sources: List<Source>
)

data class Source(
    val title: String,
    val chunk: String
)

A Retrofit interface can expose the backend:

interface RagApi {

    @POST("api/ask")
    suspend fun ask(
        @Body request: AskRequest
    ): AskResponse
}

The backend endpoint is responsible for retrieval and generation.

Keep network details outside the ViewModel:

class RagRepository(
    private val api: RagApi
) {
    suspend fun ask(
        question: String,
        conversationId: String?
    ): AskResponse {
        return api.ask(
            AskRequest(question, conversationId)
        )
    }
}

Use a UI state model:

sealed interface ChatState {
    data object Idle : ChatState
    data object  : ChatState
    data class Success(val response: AskResponse) : ChatState
    data class Error(val message: String) : ChatState
}

Then:

fun ask(question: String) {
    viewModelScope.launch {
        state.value = ChatState.

        try {
            val response = repository.ask(
                question,
                conversationId
            )

            state.value = ChatState.Success(response)
        } catch (e: Exception) {
            state.value = ChatState.Error(
                e.message ?: "Request failed"
            )
        }
    }
}

The backend ingestion pipeline might look like:

PDF / Markdown / HTML
       ↓
Text Extraction
       ↓
Chunking
       ↓
Embedding Model
       ↓
Vector Database

A chunk might contain:

{
  "text": "Kotlin coroutines provide structured concurrency...",
  "document": "kotlin-guide.md",
  "section": "Coroutines"
}

The metadata is useful when displaying citations to the user.

An embedding model converts text into a vector.

Conceptually:

"Kotlin coroutines"
        ↓
[0.021, -0.41, 0.73, ...]

The vector database stores this representation and supports similarity search.

When the user asks:

How does structured concurrency work?

The backend generates an embedding for the question and searches for similar chunks.

The top results are then added to the LLM prompt.

A simplified prompt could be:

Use the following context to answer the question.

Context:
{retrieved_chunks}

Question:
{user_question}

A useful RAG assistant should return sources rather than only an answer.

For example:

LazyColumn {
    items(response.sources) { source ->
        Text(source.title)
        Text(source.chunk)
    }
}

This gives users a way to verify the generated answer.

For a better chat experience, the backend can stream generated tokens.

Depending on your backend protocol, Kotlin can consume Server-Sent Events or another streaming protocol.

The UI can then append chunks as they arrive:

_response.update { current ->
    current + token
}

A RAG system should not blindly answer every question.

If retrieval produces weak matches, the backend can return:

I could not find enough information in the provided documents.

This is often safer than allowing the model to invent an answer.

Never put LLM API keys or vector database credentials directly into an Android application.

Use:

Android App
    ↓ HTTPS
Backend API
    ↓
Vector Database
    ↓
LLM Provider

Authenticate mobile users at the API layer and enforce authorization when retrieving private documents.

Once the basic pipeline works, add:

A Kotlin RAG assistant is more than an AI chat screen. The Android application should focus on user experience and secure API communication, while the backend manages embeddings, retrieval, document permissions, and LLM orchestration.

This architecture scales much better than embedding AI credentials and vector database logic directly inside the mobile application.

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

source & further reading

dev.to — original article Your MCP server didn't break. The protocol did. I built a RAG assistant, then found out my architecture change made it worse Building Kisan Mitra: How I Built an Ultra-Fast Voice AI for Indian Farmers in 10 Days