{"slug": "build-an-on-device-llm-chatbot-with-kotlin-and-tensorflow-lite", "title": "Build an On-Device LLM Chatbot with Kotlin and TensorFlow Lite", "summary": "A developer detailed the architecture for building an on-device LLM chatbot using Kotlin and TensorFlow Lite, focusing on mobile integration, model execution, prompt handling, and performance. The tutorial covers creating a model runner, tokenization, off-main-thread inference, and incremental token generation to keep the UI responsive.", "body_md": "Large language models are usually accessed through cloud APIs, but modern Android devices can also run smaller AI models locally. This makes it possible to build applications that work offline and keep sensitive prompts on the device.\n\nIn this tutorial, we will design the architecture of an on-device LLM chatbot using Kotlin and TensorFlow Lite. The focus is on the mobile integration layer, model execution, prompt handling, and performance considerations.\n\nThe application will have:\n\nThe exact model and tokenizer implementation depends on the model architecture you choose. Always use a model converted and packaged for the runtime supported by your Android application.\n\nA simple architecture looks like this:\n\n```\nChat UI\n   |\nViewModel\n   |\nLLM Repository\n   |\nTokenizer\n   |\nTensorFlow Lite Interpreter\n   |\nLocal Model\n```\n\nKeeping model execution behind a repository makes it easier to replace the model later.\n\nCreate an Android project with Kotlin and add TensorFlow Lite dependencies appropriate for the runtime and model you selected.\n\nFor example:\n\n```\ndependencies {\n    implementation(\"org.tensorflow:tensorflow-lite:<version>\")\n}\n```\n\nUse the current compatible TensorFlow Lite version rather than copying an old version number from a tutorial.\n\nPlace your model in:\n\n```\napp/src/main/assets/\n```\n\nFor example:\n\n```\nassets/\n└── model.tflite\n```\n\nCreate a small model runner responsible for loading the TensorFlow Lite interpreter.\n\n```\nclass LlmRunner(\n    private val context: Context\n) {\n    private val interpreter: Interpreter by lazy {\n        val model = loadModel(\"model.tflite\")\n        Interpreter(model)\n    }\n\n    private fun loadModel(name: String): MappedByteBuffer {\n        val fileDescriptor = context.assets.openFd(name)\n\n        FileInputStream(fileDescriptor.fileDescriptor).use { input ->\n            return input.channel.map(\n                FileChannel.MapMode.READ_ONLY,\n                fileDescriptor.startOffset,\n                fileDescriptor.declaredLength\n            )\n        }\n    }\n}\n```\n\nThe model runner should not be called directly from the main thread.\n\nLLMs operate on tokens rather than normal strings. Your tokenizer must convert the user's prompt into the integer representation expected by the model.\n\nConceptually:\n\n```\nval prompt = \"Explain Kotlin coroutines\"\nval tokens = tokenizer.encode(prompt)\n```\n\nAfter inference, the generated token IDs need to be decoded back into text.\n\n```\nval text = tokenizer.decode(outputTokens)\n```\n\nThe tokenizer must match the model. Using an incompatible tokenizer can produce invalid input or meaningless output.\n\nInference can be computationally expensive, so use a coroutine dispatcher designed for CPU work.\n\n```\nclass ChatViewModel(\n    private val runner: LlmRunner\n) : ViewModel() {\n\n    fun generate(prompt: String) {\n        viewModelScope.launch {\n            val result = withContext(Dispatchers.Default) {\n                runner.generate(prompt)\n            }\n\n            // Update UI state here\n        }\n    }\n}\n```\n\nThis prevents long inference operations from blocking Android's UI thread.\n\nA production chatbot should avoid waiting unnecessarily before showing output. Depending on the model runtime, you can expose generated tokens or chunks as they become available.\n\nA simplified API could look like:\n\n```\ninterface LlmRunner {\n    suspend fun generate(\n        prompt: String,\n        onToken: (String) -> Unit\n    )\n}\n```\n\nThen the ViewModel can update the chat state incrementally.\n\n``` php\nrunner.generate(prompt) { token ->\n    _response.update { it + token }\n}\n```\n\nThe actual implementation depends on whether the selected model/runtime supports incremental generation.\n\nSending the complete conversation to the model on every request increases the token count.\n\nKeep a bounded history:\n\n```\ndata class ChatMessage(\n    val role: String,\n    val content: String\n)\n```\n\nBefore inference, construct a prompt from only the relevant recent messages.\n\nFor example:\n\n```\nval recentMessages = messages.takeLast(10)\n```\n\nFor larger applications, consider summarizing older messages instead of keeping everything.\n\nOn-device models can consume significant memory. Quantization can reduce model size and sometimes improve inference performance.\n\nCommon approaches include:\n\nThe best option depends on the model and target device.\n\nYou should benchmark:\n\nNever execute model inference inside a click listener directly:\n\n```\nbutton.setOnClickListener {\n    runner.generate(prompt)\n}\n```\n\nInstead, move the work into a coroutine or another background execution mechanism.\n\nThis is especially important for larger models.\n\nLocal inference can fail because of:\n\nWrap model execution with appropriate error handling and expose useful UI states:\n\n```\nsealed interface ChatState {\n    data object Idle : ChatState\n    data object Generating : ChatState\n    data class Success(val text: String) : ChatState\n    data class Error(val message: String) : ChatState\n}\n```\n\nOne advantage of local inference is that prompts do not need to leave the device. However, the model itself is part of your application package and may be extracted.\n\nAvoid embedding secrets in the model or APK.\n\nAn on-device LLM chatbot combines Kotlin application development with model inference, tokenization, concurrency, and mobile optimization.\n\nThe most important production lesson is that AI inference should be treated as a resource-intensive workload. Model size, memory consumption, latency, and thermal behavior all matter on mobile devices.\n\nOnce the basic architecture works, you can extend it with conversation memory, retrieval-augmented generation, voice input, function calling, or multimodal models.\n\nSDK Flutter: [https://github.com/v-modal/vmodal_sdk_flutter](https://github.com/v-modal/vmodal_sdk_flutter)\n\nSDK Android: [https://github.com/v-modal/vmodal_sdk_android](https://github.com/v-modal/vmodal_sdk_android)\n\nDiscord: [https://discord.gg/K72z28KUx](https://discord.gg/K72z28KUx)", "url": "https://wpnews.pro/news/build-an-on-device-llm-chatbot-with-kotlin-and-tensorflow-lite", "canonical_source": "https://dev.to/vmodal_ai/build-an-on-device-llm-chatbot-with-kotlin-and-tensorflow-lite-36d3", "published_at": "2026-08-14 19:06:39+00:00", "updated_at": "2026-08-14 19:35:45.689160+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "artificial-intelligence"], "entities": ["Kotlin", "TensorFlow Lite", "Android"], "alternates": {"html": "https://wpnews.pro/news/build-an-on-device-llm-chatbot-with-kotlin-and-tensorflow-lite", "markdown": "https://wpnews.pro/news/build-an-on-device-llm-chatbot-with-kotlin-and-tensorflow-lite.md", "text": "https://wpnews.pro/news/build-an-on-device-llm-chatbot-with-kotlin-and-tensorflow-lite.txt", "jsonld": "https://wpnews.pro/news/build-an-on-device-llm-chatbot-with-kotlin-and-tensorflow-lite.jsonld"}}