Build an On-Device LLM Chatbot with Kotlin and TensorFlow Lite

A developer detailed the architecture for building an on-device LLM chatbot using Kotlin and TensorFlow Lite, focusing on mobile integration, model execution, prompt handling, and performance. The tutorial covers creating a model runner, tokenization, off-main-thread inference, and incremental token generation to keep the UI responsive.

Large language models are usually accessed through cloud APIs, but modern Android devices can also run smaller AI models locally. This makes it possible to build applications that work offline and keep sensitive prompts on the device. In this tutorial, we will design the architecture of an on-device LLM chatbot using Kotlin and TensorFlow Lite. The focus is on the mobile integration layer, model execution, prompt handling, and performance considerations. The application will have: The exact model and tokenizer implementation depends on the model architecture you choose. Always use a model converted and packaged for the runtime supported by your Android application. A simple architecture looks like this: Chat UI | ViewModel | LLM Repository | Tokenizer | TensorFlow Lite Interpreter | Local Model Keeping model execution behind a repository makes it easier to replace the model later. Create an Android project with Kotlin and add TensorFlow Lite dependencies appropriate for the runtime and model you selected. For example: dependencies { implementation "org.tensorflow:tensorflow-lite:<version " } Use the current compatible TensorFlow Lite version rather than copying an old version number from a tutorial. Place your model in: app/src/main/assets/ For example: assets/ └── model.tflite Create a small model runner responsible for loading the TensorFlow Lite interpreter. class LlmRunner private val context: Context { private val interpreter: Interpreter by lazy { val model = loadModel "model.tflite" Interpreter model } private fun loadModel name: String : MappedByteBuffer { val fileDescriptor = context.assets.openFd name FileInputStream fileDescriptor.fileDescriptor .use { input - return input.channel.map FileChannel.MapMode.READ ONLY, fileDescriptor.startOffset, fileDescriptor.declaredLength } } } The model runner should not be called directly from the main thread. LLMs operate on tokens rather than normal strings. Your tokenizer must convert the user's prompt into the integer representation expected by the model. Conceptually: val prompt = "Explain Kotlin coroutines" val tokens = tokenizer.encode prompt After inference, the generated token IDs need to be decoded back into text. val text = tokenizer.decode outputTokens The tokenizer must match the model. Using an incompatible tokenizer can produce invalid input or meaningless output. Inference can be computationally expensive, so use a coroutine dispatcher designed for CPU work. class ChatViewModel private val runner: LlmRunner : ViewModel { fun generate prompt: String { viewModelScope.launch { val result = withContext Dispatchers.Default { runner.generate prompt } // Update UI state here } } } This prevents long inference operations from blocking Android's UI thread. A production chatbot should avoid waiting unnecessarily before showing output. Depending on the model runtime, you can expose generated tokens or chunks as they become available. A simplified API could look like: interface LlmRunner { suspend fun generate prompt: String, onToken: String - Unit } Then the ViewModel can update the chat state incrementally. php runner.generate prompt { token - response.update { it + token } } The actual implementation depends on whether the selected model/runtime supports incremental generation. Sending the complete conversation to the model on every request increases the token count. Keep a bounded history: data class ChatMessage val role: String, val content: String Before inference, construct a prompt from only the relevant recent messages. For example: val recentMessages = messages.takeLast 10 For larger applications, consider summarizing older messages instead of keeping everything. On-device models can consume significant memory. Quantization can reduce model size and sometimes improve inference performance. Common approaches include: The best option depends on the model and target device. You should benchmark: Never execute model inference inside a click listener directly: button.setOnClickListener { runner.generate prompt } Instead, move the work into a coroutine or another background execution mechanism. This is especially important for larger models. Local inference can fail because of: Wrap model execution with appropriate error handling and expose useful UI states: sealed interface ChatState { data object Idle : ChatState data object Generating : ChatState data class Success val text: String : ChatState data class Error val message: String : ChatState } One advantage of local inference is that prompts do not need to leave the device. However, the model itself is part of your application package and may be extracted. Avoid embedding secrets in the model or APK. An on-device LLM chatbot combines Kotlin application development with model inference, tokenization, concurrency, and mobile optimization. The most important production lesson is that AI inference should be treated as a resource-intensive workload. Model size, memory consumption, latency, and thermal behavior all matter on mobile devices. Once the basic architecture works, you can extend it with conversation memory, retrieval-augmented generation, voice input, function calling, or multimodal models. SDK Flutter: https://github.com/v-modal/vmodal sdk flutter https://github.com/v-modal/vmodal sdk flutter SDK Android: https://github.com/v-modal/vmodal sdk android https://github.com/v-modal/vmodal sdk android Discord: https://discord.gg/K72z28KUx https://discord.gg/K72z28KUx