Large language models are usually accessed through cloud APIs, but modern Android devices can also run smaller AI models locally. This makes it possible to build applications that work offline and keep sensitive prompts on the device.
In this tutorial, we will design the architecture of an on-device LLM chatbot using Kotlin and TensorFlow Lite. The focus is on the mobile integration layer, model execution, prompt handling, and performance considerations.
The application will have:
The exact model and tokenizer implementation depends on the model architecture you choose. Always use a model converted and packaged for the runtime supported by your Android application.
A simple architecture looks like this:
Chat UI
|
ViewModel
|
LLM Repository
|
Tokenizer
|
TensorFlow Lite Interpreter
|
Local Model
Keeping model execution behind a repository makes it easier to replace the model later.
Create an Android project with Kotlin and add TensorFlow Lite dependencies appropriate for the runtime and model you selected.
For example:
dependencies {
implementation("org.tensorflow:tensorflow-lite:<version>")
}
Use the current compatible TensorFlow Lite version rather than copying an old version number from a tutorial.
Place your model in:
app/src/main/assets/
For example:
assets/
└── model.tflite
Create a small model runner responsible for the TensorFlow Lite interpreter.
class LlmRunner(
private val context: Context
) {
private val interpreter: Interpreter by lazy {
val model = loadModel("model.tflite")
Interpreter(model)
}
private fun loadModel(name: String): MappedByteBuffer {
val fileDescriptor = context.assets.openFd(name)
FileInputStream(fileDescriptor.fileDescriptor).use { input ->
return input.channel.map(
FileChannel.MapMode.READ_ONLY,
fileDescriptor.startOffset,
fileDescriptor.declaredLength
)
}
}
}
The model runner should not be called directly from the main thread.
LLMs operate on tokens rather than normal strings. Your tokenizer must convert the user's prompt into the integer representation expected by the model.
Conceptually:
val prompt = "Explain Kotlin coroutines"
val tokens = tokenizer.encode(prompt)
After inference, the generated token IDs need to be decoded back into text.
val text = tokenizer.decode(outputTokens)
The tokenizer must match the model. Using an incompatible tokenizer can produce invalid input or meaningless output.
Inference can be computationally expensive, so use a coroutine dispatcher designed for CPU work.
class ChatViewModel(
private val runner: LlmRunner
) : ViewModel() {
fun generate(prompt: String) {
viewModelScope.launch {
val result = withContext(Dispatchers.Default) {
runner.generate(prompt)
}
// Update UI state here
}
}
}
This prevents long inference operations from blocking Android's UI thread.
A production chatbot should avoid waiting unnecessarily before showing output. Depending on the model runtime, you can expose generated tokens or chunks as they become available.
A simplified API could look like:
interface LlmRunner {
suspend fun generate(
prompt: String,
onToken: (String) -> Unit
)
}
Then the ViewModel can update the chat state incrementally.
runner.generate(prompt) { token ->
_response.update { it + token }
}
The actual implementation depends on whether the selected model/runtime supports incremental generation.
Sending the complete conversation to the model on every request increases the token count.
Keep a bounded history:
data class ChatMessage(
val role: String,
val content: String
)
Before inference, construct a prompt from only the relevant recent messages.
For example:
val recentMessages = messages.takeLast(10)
For larger applications, consider summarizing older messages instead of keeping everything.
On-device models can consume significant memory. Quantization can reduce model size and sometimes improve inference performance.
Common approaches include:
The best option depends on the model and target device.
You should benchmark:
Never execute model inference inside a click listener directly:
button.setOnClickListener {
runner.generate(prompt)
}
Instead, move the work into a coroutine or another background execution mechanism.
This is especially important for larger models.
Local inference can fail because of:
Wrap model execution with appropriate error handling and expose useful UI states:
sealed interface ChatState {
data object Idle : ChatState
data object Generating : ChatState
data class Success(val text: String) : ChatState
data class Error(val message: String) : ChatState
}
One advantage of local inference is that prompts do not need to leave the device. However, the model itself is part of your application package and may be extracted.
Avoid embedding secrets in the model or APK.
An on-device LLM chatbot combines Kotlin application development with model inference, tokenization, concurrency, and mobile optimization.
The most important production lesson is that AI inference should be treated as a resource-intensive workload. Model size, memory consumption, latency, and thermal behavior all matter on mobile devices.
Once the basic architecture works, you can extend it with conversation memory, retrieval-augmented generation, voice input, function calling, or multimodal models.
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx