If this project helped you, please ⭐️ star it. Also try Box — a full-stack on-device AI app built on the same philosophy.

📱 Screenshots

• 100% Offline — no INTERNET permission in the manifest, cannot phone home
• On-Device Inference — GGUF models via llama.cpp; runtime CPU dispatch picks the best kernel set (dotprod / fp16 / i8mm / SVE) for your exact SoC
• GPU Acceleration (Vulkan) — opt-in toggle in Settings with per-layer offload control and automatic CPU fallback; detects and names your GPU
• Fast Multi-Turn Chat — incremental prompt processing: each turn feeds only the new message into the KV cache instead of re-processing the whole conversation
• Performance Controls — CPU thread count, prompt-phase threading across all cores, memory-mapped loading, RAM lock, experimental quantized KV cache
• Streaming Responses — token-by-token output as the model generates
• Import Any Model — bring your own GGUF at runtime via file picker
• Multiple Conversations — auto-titled, renameable, searchable
• Translator — 75+ languages
• Advanced Sampling — Temperature, Top-P, Top-K, Min-P, Repeat Penalty
• System Prompts — General, Coder, Creative Writer, Tutor, Translator
• Markdown + TTS — formatted responses, read aloud via system TTS
• Thinking Tag Stripping — hides <think> blocks from reasoning models
• Theming — 11 Ptyxis terminal palettes (Cobalt Neon default) + Catppuccin (all 4 flavors × 14 accents) + Dracula (7 accents) + System / Light / Dark / AMOLED with Material You accents, plus monochrome-accent mode
• 13 Bundled Fonts — from Turret Road (default) to IBM Plex, Playfair Display, and Press Start 2P, with an app-wide text-size slider
• Context Bar — live token-usage indicator on the chat screen
• Tamper Detection — release builds verify the APK signing certificate at startup and refuse to run if repackaged
• Security — encrypted settings, optional biometric lock, secure file deletion
• Chat Backup — export/import as JSON
• Gemma 4 — native chat-template support, including the elastic E2B/E4B models with shared-KV layers
• Actionable Errors — model-load failures surface the real llama.cpp reason instead of a generic message

Install

v5.1.0 ships as a single Vanilla APK — bring your own GGUF model and import it from Settings.

Requires Android 14+ and arm64-v8a. Vast majority of Android devices since 2019 are arm64.

1. Download from Releases
2. Settings → Apps → Install unknown apps → allow your file manager
3. Open the APK, tap Install, complete onboarding
4. Settings → Model → Import GGUF Model (download one from HuggingFace)

Or via ADB:

adb install OfflineLLM_V5.1.0_Signed_Release_Vanilla.apk

Tamper detection: release builds verify the APK signing certificate at launch. The app exits with an "Unverified App" dialog if anyone has re-signed the APK with a different key.

Performance

• CPU: the APK bundles seven ggml-cpu kernel variants (armv8.0 → armv9.2); at load time ggml scores them against your CPU's features and loads the fastest one. Prompt processing additionally uses every core, while generation sticks to the big cores.
• GPU: Settings → Performance → GPU Acceleration (Vulkan). Biggest wins on Adreno-class GPUs and anything with cooperative-matrix support; Mali midrange may tie the CPU. If a GPU load fails, the app automatically retries on CPU.
• Long chats: turn 2 onward only processes your new message — no more re-crunching the whole conversation each turn.
• All performance settings apply the next time a model is loaded.

Recommended Models

Model (Q4_K_M)	Approx. Size	RAM Required / Best For
gemma-3-270m-it-qat-Q4_K_M.gguf	~300 MB	2–4 GB RAM devices, fast responses
Qwen3.5 0.8B Q4_K_M	~530 MB	Good balance for 4–6 GB RAM
gemma-4-E2B-it-GGUF (2.3B effective)	~1.3 GB	Recommended for 6–8 GB RAM
gemma-4-E4B-it-GGUF (4.5B effective)	~2.5 GB	Recommended for 8 GB RAM
Qwen3.5 4B Q4_K_M	~2.5 GB	Flagship (12 GB+ RAM)

Search the model name + "GGUF" on HuggingFace. Q4_K_M is the best quality/speed balance.

Build from Source

Prerequisites: JDK 17, Android SDK (compileSdk 37), NDK 27.2, CMake 3.22.1, a host C/C++ compiler (gcc/g++, used to build llama.cpp's Vulkan shader generator)

The Vulkan backend needs two Khronos header repos checked out next to the project directory:

git clone --recurse-submodules https://github.com/jegly/OfflineLLM.git

# Khronos headers for the Vulkan GPU backend (siblings of the project dir)
git clone https://github.com/KhronosGroup/Vulkan-Headers.git
git clone https://github.com/KhronosGroup/SPIRV-Headers.git
cmake -S SPIRV-Headers -B SPIRV-Headers/build -DCMAKE_INSTALL_PREFIX=SPIRV-Headers/install
cmake --install SPIRV-Headers/build

cd OfflineLLM

# Optional: bundle a model in the APK
cp /path/to/model.gguf app/src/main/assets/model/

./gradlew assembleDebug

First build compiles llama.cpp from source, including 1,400 Vulkan compute shaders and seven CPU-variant libraries (20–30 min). Subsequent builds are fast.

Project structure

• smollm/ — Native llama.cpp JNI module
  • src/main/cpp/ — C++ inference engine + JNI bridge
  • src/main/java/ — SmolLM.kt, GGUFReader.kt wrappers
• app/ — Main Android application (src/main/java/com/jegly/offlineLLM/)
  • ai/ — InferenceEngine, ModelManager, SystemPrompts
  • data/ — Room database, DAOs, repositories
  • di/ — Hilt dependency injection modules
  • ui/ — Compose screens, components, theme, navigation
  • utils/ — BiometricHelper, MemoryMonitor, SecurityUtils, TTS
• llama.cpp/ — git submodule

Security & Privacy

• Zero network permissions (no INTERNET, no ACCESS_NETWORK_STATE)
• No Google Play Services or Firebase dependencies
• Encrypted settings via Jetpack Security
• Optional biometric lock
• Memory Tagging Extension enabled (memtagMode="sync")
• Secure deletion — files overwritten before removal
• No logging of prompts or responses

License

Apache License 2.0. llama.cpp backend: MIT. Native wrapper adapted from SmolChat-Android (Apache 2.0).

---