English · 中文 · 日本語 · Español · Français · Italiano

Run Mixture-of-Experts models larger than the device's memory by streaming the experts from flash. Built on llama.cpp's public API; Android, iOS, macOS, Windows and Linux; a local OpenAI-compatible server on each.

12 GB Android phone	12 GB iPhone
16 GB Apple-silicon laptop

Qwen3.6-35B-A3B generating on each device, in real time, in the shipped apps. The model is larger than each device's memory.

A Mixture-of-Experts model uses a few of its experts for each token. BigMoeOnEdge keeps in memory the weights every token needs and a cache of recently used experts, and reads the other experts from flash when the router asks for them. The output is identical to running the model fully in memory; the tests check it.

• Planner. --auto measures the machine (available memory, storage speed, each CPU, GPU and NPU) and the model's header, then sets the cache size, what stays resident, and which device runs the prompt and which the answer. Its rules name no platform or device. Settings given by hand are kept.
• Models. Any MoE gguf that llama.cpp loads, in any quantization. Expert tensors are found from the file, so a new architecture needs no code here.
• Backends. CPU, Metal, Vulkan, OpenCL and Hexagon NPU, loaded as modules; CUDA, HIP and SYCL modules built with their own toolchains load the same way.
• Apps and server. Native apps on Android and iOS, a web app on the desktop, and an OpenAI-compatible server on all of them.

Performance

Decode speed, measured on the devices below. The engine sizes itself to the machine, so stronger hardware runs faster.

Model	On disk	Device	Model / RAM	Decode
DeepSeek V4 Flash 284B	91 GB	12 GB Android phone	7.6×	0.94 tok/s
Qwen3.8-Flash-Next 125B	80 GB	12 GB Android phone	6.7×	2.2-3.5 tok/s
gpt-oss-120b	60 GB	12 GB Android phone	5×	1.3 tok/s
Qwen3.6-35B-A3B	21 GB	12 GB Android phone	1.9×	4.8-5.0 tok/s
Qwen3.6-35B-A3B	21 GB	12 GB iPhone	1.9×	2.6-3.0 tok/s
Qwen3-30B-A3B	18.5 GB	12 GB Android phone	1.5×	5.2 tok/s
Gemma-4-26B-A4B	17 GB	12 GB Android phone	1.4×	4.1 tok/s
Qwen3.8-Flash-Next 125B	76 GB	16 GB Apple-silicon laptop	4.8×	6.5-9 tok/s
Qwen3.6-35B-A3B	21 GB	16 GB Apple-silicon laptop	1.3×	18-24 tok/s
Qwen3-Coder-30B-A3B	18.6 GB	16 GB Apple-silicon laptop	1.2×	23 tok/s; 7-9 inside opencode
Qwen3.6-35B-A3B	21 GB	16 GB Windows laptop	1.3×	4.8 tok/s

Configurations and method: docs/benchmarks.md. To add a machine, run scripts/bench-report.sh /path/to/model.gguf and paste the table into a benchmark report.

Quick start

Android. Install the APK from the latest release and pick a model under Models. Settings → Serve on the network shares it, with a key.

iOS. Build the app with Xcode from examples/ios. Experimental.

Desktop (macOS, Windows, Linux). Download the bmoe-desktop archive from the same release and run bmoe-server --open; the app opens at http://127.0.0.1:8765/.

Command line.

bmoe-cli -m model.gguf --auto -p "Explain MoE routing."   # measure, plan, run
bmoe-cli -m model.gguf --plan                              # print the plan and exit

OpenAI-compatible server

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8765/v1", api_key="local")
reply = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Write a haiku about experts on flash."}],
)
print(reply.choices[0].message.content)

Streaming, tool calls and reasoning are supported. For opencode, add the server as an OpenAI-compatible provider with baseURL set to http://127.0.0.1:8765/v1. To serve other machines, start the server with --host and --api-key. API reference: docs/server-api.md.

How it works

The engine uses llama.cpp's public API: every quantization, tokenizer and chat template llama.cpp supports works unchanged, and a llama.cpp upgrade is a submodule bump. The routed experts are read from the file's tensor layout (docs/which-models-stream.md). ctest checks that streamed output equals fully resident output across the cache, evictions, overlap, sessions and split models (docs/architecture.md).

The planner

The planner reads the model's shape from its header and measures the machine. The same rules give a phone and a laptop different plans. The Plan screen shows the result: where the memory goes, what each device does, and the reason for each decision. docs/hardware-planning.md.

Platforms

	Android	macOS	Windows	Linux	iOS
Engine and CLI	arm64	Apple silicon	x86_64	x86_64, aarch64	arm64 (in the app)
App and server	native app	web app	web app	web app	native app, built from source
Accelerators	Hexagon NPU, Vulkan, OpenCL	Metal	Vulkan	Vulkan	Metal
Run on hardware	12 GB phone	16 GB laptop	16 GB laptop	CI, community	12 GB iPhone (#235)

Details and what was measured where: docs/platforms.md.

Build from source

git clone --recursive https://github.com/Helldez/BigMoeOnEdge.git
cd BigMoeOnEdge
cd ui && npm ci && npm run build && cd ..
scripts/build-host.sh
build/bin/bmoe-server --open

Tests: cd build && ctest --output-on-failure (needs python3 with the gguf package). Android: scripts/build-android.ps1, then the APK in examples/android. CONTRIBUTING.md.

Documentation

docs/, indexed by task. Start with hardware-planning.md, settings.md, server-api.md, platforms.md and limitations.md.

Prior art

AirLLM, Apple's LLM in a flash, FlexGen, PowerInfer and EdgeMoE. The closest recent work is flash-moe, a Metal engine that streams a 397B MoE from SSD on Apple silicon; BigMoeOnEdge uses llama.cpp's public API across architectures and platforms.

License

Apache-2.0. See LICENSE.