{"slug": "run-local-llms-on-multiple-android-ios-phones", "title": "Run local LLMs on multiple Android, iOS phones", "summary": "A Flutter-based cross-platform AI chat client, built on the llama.cpp wrapper plugin llama_flutter_android, runs GGUF models locally on ARM64 Android phones with Vulkan GPU acceleration while falling back to OpenAI, Anthropic, Google Gemini or Kimi (Moonshot AI) cloud APIs on unsupported platforms. The app detects device RAM and GPU tier on first launch to set context size and thread count, uses 5-second idle detection and 180-second hard timeouts, and stores chats locally via Hive; image generation was tested on the Moto G71, OnePlus 10R, Pixel 6A, Poco F1 and Samsung S23, with a 4-step fast mode and a 20-step run generated on the Pixel 6. A structured multi-step task view is listed as in development and not yet available in the app.", "body_md": "A production-ready, cross-platform AI chat client built with Flutter. It unifies local on-device LLM inference (Android) with cloud API access, giving users full control over how their models run.\n\n*Image generation tested on Moto G71 (Snapdragon), Oneplus 10r (Mediatek), Pixel 6A (Tensor), Poco F1 (Snapdragon), Samsung s23 (Snapdragon) 4 steps fast*\n\n*Generated on pixel 6 with 20 step*\n\n- **Local Inference on Android** — Download and run GGUF models directly on your phone using GPU-accelerated inference (Vulkan). No internet required after download.\n- **Cloud API Fallback** — Seamlessly switch to OpenAI, Anthropic, Google Gemini, or Kimi (Moonshot AI) when you need more power or are on unsupported platforms.\n- **Multimodal Chat** — Send text and images in conversations. Vision support works with both local models (Qwen2-VL) and cloud providers.\n- **Persistent Sessions** — All chats and settings are stored locally via Hive. Nothing leaves your device unless you explicitly choose cloud mode.\n- **Background Services** — Firebase Cloud Messaging integration for push updates and background task handling.\n- **Smart Auto-Configuration** — On first launch, the app detects your device's RAM and recommends optimal context size and token limits automatically.\n- **Task Management (in development)** — A task view for structured, multi-step AI workflows is being built. It is not available in the app yet.\n\n- **Framework:** Flutter 3.x (Dart >=3.3.0)\n- **State Management:** GetX\n- **Local Storage:** Hive\n- **Networking:** Dio +`package:http`\n- **Background Execution:**`flutter_background_service` +`flutter_local_notifications`\n- **Push Notifications:** Firebase Core + Firebase Messaging\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│                        UI Layer                              │\n│   ChatView / ModelView / ServerView / SettingsView          │\n└──────────────────────────┬──────────────────────────────────┘\n                           │\n┌──────────────────────────▼──────────────────────────────────┐\n│                    Controllers (GetX)                        │\n│   ChatController · TaskController · ModelController         │\n│   SettingsController · HomeController                       │\n└──────────────────────────┬──────────────────────────────────┘\n                           │\n┌──────────────────────────▼──────────────────────────────────┐\n│                      Services                                │\n│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │\n│  │ InferenceService│  │  CloudService   │  │DownloadSvc  │ │\n│  │  (local GGUF)   │  │ (OpenAI/Claude/ │  │ (model dl)  │ │\n│  │                 │  │  Gemini/Kimi)   │  │             │ │\n│  └─────────────────┘  └─────────────────┘  └─────────────┘ │\n│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────┐ │\n│  │  HiveService    │  │ DeviceInfoSvc   │  │ExecutionSvc │ │\n│  │  ( persistence) │  │  (RAM/GPU tier) │  │ (bg tasks)  │ │\n│  └─────────────────┘  └─────────────────┘  └─────────────┘ │\n└─────────────────────────────────────────────────────────────┘\n```\n\nThe app uses `llama_flutter_android`, a custom Flutter plugin wrapping `llama.cpp` for ARM64 devices. At runtime it:\n\n1. **Detects GPU capabilities** via Vulkan to determine offload layers.\n2. **Selects thread count** based on device tier (ultra / high / mid / low).\n3. **Loads the GGUF model** with progress streaming.\n4. **Generates tokens** via`generateChat()` with native chat-template support (ChatML, Llama-3, Gemma, Phi).\n5. **Falls back** to manual prompt construction if native templates fail.\n\nIdle detection (5s) and hard timeouts (180s) keep the UX responsive even on underpowered hardware.\n\n`CloudService` normalizes four different API shapes into a single interface:\n\n- **OpenAI** — standard`/v1/chat/completions`\n- **Anthropic** — Messages API with separate system param\n- **Google Gemini** —`generateContent` with inline image base64\n- **Kimi** — OpenAI-compatible endpoint from Moonshot AI\n\nAPI keys are stored in Hive and never transmitted anywhere except to the provider's endpoint.\n\nLocal inference is conditionally compiled:\n\n- **Android** →`inference_android.dart` (full llama.cpp engine)\n- **Web** →`inference_stub.dart` (cloud-only, local coming soon)\n- **iOS** →`inference_android.dart` (full llama.cpp engine via Metal GPU)\n\nThe `InferenceService` exposes `supportsLocalInference` so the UI can hide local-model UI on unsupported platforms.\n\n| Platform | Local Inference | Cloud APIs | Notes | \n|---|---|---|---|\n| Android | ✅ Yes | ✅ Yes | CPU offload via NEON; minSdk 28 | \n| iOS | ✅ Yes | ✅ Yes | Metal GPU acceleration | \n| Web | ❌ No | ✅ Yes | Cloud-only (local coming soon) | \n\nThe iOS/iPad release is distributed as a sideloadable IPA file. Due to the RAM requirements for local models, iPad is the recommended iOS target, but it can be installed on iPhones.\n\nHere is how you can install the app using AltStore and AltServer:\n\n**Prerequisites:**\n\n- Ensure **AltServer** is installed and running on your Mac or Windows PC.\n- Your iOS device must be connected to the same Wi-Fi network as your computer, or connected directly via a USB cable.\n- You must have **AltStore** successfully installed on your iOS device.\n\n**Installation Steps:**\n\n1. **Download the IPA:** Go to the[Releases](https://github.com/orailnoor/cross-platform-llm-client/releases) page on your iOS device (using Safari) and download the latest`PrivateLM-iOS.zip` package, extract it to get the`.ipa` file.\n2. **Open in AltStore:**  - Locate the extracted `.ipa` file in your device's \"Files\" app.\n  - Tap the **Share** button and select**AltStore** from the list of apps.\n  - *Alternatively* , you can open the AltStore app, go to the**My Apps** tab, tap the**\"+\" (plus)** icon in the top-left corner, and select the downloaded IPA file.\n3. Locate the extracted \n4. **Install:** AltStore will begin installing the app. You may be prompted to enter your Apple ID credentials if this is your first time.\n5. **Enable Developer Mode:** On iOS 16 and later, ensure \"Developer Mode\" is enabled in your device's Settings (`Settings > Privacy & Security > Developer Mode` ) to launch the sideloaded app.\n6. **Launch:** Once installation is finished, PrivateLM will appear in the**My Apps** tab in AltStore and on your device's home screen!\n\n*Note: Sideloaded apps via AltStore expire after 7 days, but AltStore will attempt to automatically refresh them in the background when connected to the same Wi-Fi as your computer running AltServer.*\n\n- Flutter SDK >=3.3.0\n- Android SDK (API 26+)\n- JDK 17\n- NDK (bundled with Android SDK)\n\n```\nflutter pub get\nflutter build apk --debug\n```\n\nRelease APKs require a stable signing key. Copy\n`android/key.properties.example` to `android/key.properties`, fill in the\nkeystore values, and keep both the key and its backup. Android accepts an APK\nupgrade only when it is signed with the same key as the installed APK.\n\n```\ncp android/key.properties.example android/key.properties\nflutter build apk --release --split-per-abi\n```\n\nNever rotate the signing key between GitHub releases. The per-ABI APKs must\nalso keep increasing the build number in `pubspec.yaml`.\n\n```\nflutter pub get\ncd ios\npod install\nflutter build ios\nflutter pub get\nflutter build web --release\n```\n\nMIT — see [LICENSE](https://github.com/orailnoor/cross-platform-llm-client/blob/main/LICENSE) for details.", "url": "https://wpnews.pro/news/run-local-llms-on-multiple-android-ios-phones", "canonical_source": "https://github.com/orailnoor/cross-platform-llm-client", "published_at": "2026-11-06 05:44:16+00:00", "updated_at": "2026-11-06 06:20:49.404345+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-products", "ai-infrastructure"], "entities": ["Flutter", "llama.cpp", "llama_flutter_android", "OpenAI", "Anthropic", "Google Gemini", "Kimi", "Moonshot AI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/run-local-llms-on-multiple-android-ios-phones", "markdown": "https://wpnews.pro/news/run-local-llms-on-multiple-android-ios-phones.md", "text": "https://wpnews.pro/news/run-local-llms-on-multiple-android-ios-phones.txt", "jsonld": "https://wpnews.pro/news/run-local-llms-on-multiple-android-ios-phones.jsonld"}}