On-device AI toolkit enabling LLM chat, streaming, speech-to-text, text-to-speech and full voice-assistant pipelines—offline, private, model download/progress, lightweight neural voices and GGUF model support.
One SDK. Every device.
LLMs, vision, speech, voice agents, RAG, embeddings, and image generation, running locally on
phones, browsers, desktops, and servers.
Private by default. Offline by design. Accelerated by whatever silicon the device has.
Eight SDKs, one C++ core. A capability registry routes every call to the best engine on the device, and the Console manages your fleet from above.
Every capability below runs fully on-device behind one semantic API across the eight SDKs.
Call RunAnywhere.capabilities() (v4) to discover what the current package and device can
actually execute — enum presence alone does not mean an engine is installed.
Your code rarely picks hardware. Engines register what they can run, and the highest-priority engine that fits the device wins: QHexRT on the Snapdragon Hexagon NPU, MLX on Apple silicon, llama.cpp everywhere (Metal on Apple, CUDA on NVIDIA as an opt-in build, WebGPU in the browser), sherpa + ONNX for speech and embeddings, and Core ML for diffusion. LiteRT and ExecuTorch are reserved framework values only — they are not integrated runtimes yet.
The fastest way to feel it. Install, load, generate, all local:
pip install runanywhere
import runanywhere as ra
from runanywhere import LlmOptions
ra.initialize()
# downloads on first use
print(ra.llm.generate("Explain on-device AI in one sentence.",
LlmOptions(model="qwen2.5-0.5b")).text)
Prefer a terminal? The same core ships as a CLI:
brew install runanywhereai/tap/rcli
rcli run qwen3 "Explain on-device AI in one sentence."
Building for mobile, web, or desktop? Every platform below speaks the same API.
All SDKs ship on one version line, currently 0.20.11, from a single C++ core. Pin the same version across the core package and its backends. See Releases for what is published today.
* Web RAG may be limited to one session per process — check capabilities().rag.multiSession.
Stub / Gated / Partial mean the verb is absent, preflight-fails, or only partially wired; call capabilities() for the installed build.
| Hexagon NPU (QHexRT) | n/a | Yes | Yes | Yes | n/a | n/a | n/a | n/a |
| MLX (Apple silicon) | Yes | n/a | Yes | Yes | n/a | n/a | n/a | Yes |
| OpenAI-compatible server | n/a | n/a | n/a | n/a | n/a | n/a | Yes | Yes |
| Model download + progress | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| | Host (macOS) / Client (iOS, iPadOS) | Client | — | — | — | — | — | — |
Connect lets a macOS Swift app host a loaded language model on the local network so iOS, iPadOS, and Android clients can discover it and stream generation without downloading that model. It is app-scoped (lives with the host app process), not an OS daemon.
| Role | Supported today | Not in this release |
|---|---|---|
| Host | macOS (Swift example / SDK) | Windows, Electron, Web, RN, Flutter |
CUA on Web is "API only": the prompt/parse scaffold ships, but the catalogued Fara1.5-4B does not fit the 4 GB WASM32 heap, so no CUA model is seeded there.
Every SDK is a thin binding over runanywhere-commons, a single C++ core behind a pure C ABI. Engines plug into a capability registry and declare, per modality, what they can run. At inference time the highest-priority engine that serves the modality on the current device wins. Same code, different silicon, no branching in your app.
MetalRT, RunAnywhere's proprietary GPU inference engine for Apple silicon, powers RCLI, our on-device voice assistant for macOS with local RAG and 40+ system actions at sub-200 ms latency. Signed binaries live at metalrt-binaries.
QHexRT is RunAnywhere's inference runtime for the Qualcomm Hexagon NPU. It runs LLM, vision, speech, and text-to-speech models directly on the Snapdragon NPU (Hexagon v75 / v79 / v81) and ships as a built-in accelerator: your app calls the same loadModel and generate, and it uses the NPU automatically on supported devices.
Measured on a Samsung Galaxy S25 (Snapdragon 8 Elite, Hexagon v79):
Available on the Kotlin, Flutter, and React Native SDKs. Snapdragon (Android arm64) only.
The Python SDK and rcli both expose the local runtime as a drop-in OpenAI API, so anything that speaks the OpenAI client works against models running on your machine:
pip install "runanywhere[server]"
runanywhere serve # http://127.0.0.1:8000
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
reply = client.chat.completions.create(
model="qwen2.5-0.5b",
messages=[{"role": "user", "content": "Hello from the edge."}],
)
Endpoints: /v1/chat/completions (streaming and non-streaming, text and vision), /v1/completions, /v1/embeddings, /v1/audio/transcriptions, /v1/audio/speech, and /v1/models. rcli serve offers the same on port 8080.
The Console is the control plane for on-device AI fleets. SDKs authenticate with an API key, register the device with its hardware profile, pull their assigned models, and report per-modality telemetry.
The Console is optional. Every SDK runs fully offline without an API key, and telemetry is scoped per modality when enabled.
Prebuilt bundles published on Hugging Face; the SDK downloads the one matching the device.
Browse all models on Hugging Face
Anything not in the catalog can be pulled straight from Hugging Face or a direct URL; the core infers format, framework, and category from the artifact.
Full consumer-assistant apps, one per platform, all built on the SDK. The iOS, Android, Web, and Electron apps live in their own repositories; the Flutter and React Native ones are still in this tree.
The Android, Flutter, and React Native apps include an NPU section that detects the device's Hexagon arch and runs LLM, vision, speech, and text-to-speech on the NPU.
Minimal examples, the in-repo harnesses SDK contributors use to check a change end to end. Each builds the SDK from local source, so an edit shows up without a publish step:
Starters, minimal projects to copy from: Swift · Kotlin · Flutter · React Native · Web
Real projects built on the stack:
Business logic lives in the C++ core, so one fix lands on all eight SDKs at once.
Hexagon NPU: Snapdragon with Hexagon v75 / v79 / v81, Android arm64. MLX: Apple silicon, physical devices. Memory: 2 GB minimum, 4 GB+ recommended for larger models.
We welcome contributions. See the Contributing Guide for setup and conventions.
git clone https://github.com/RunanywhereAI/runanywhere-sdks.git
cd runanywhere-sdks
# Doctor / setup helpers
./run doctor
./run setup
# Build the native XCFrameworks into bindings/swift/Binaries/.
# Required for local Swift development.
./bindings/swift/scripts/build-core-xcframework.sh
# Stream one completion through the minimal Swift harness
cd bindings/swift/example
RUNANYWHERE_USE_LOCAL_NATIVES=1 swift run
RunAnywhere License (Apache 2.0 based, with additional commercial-use terms). See LICENSE for details.
generateStructured + enforcement mode)SpeechHandle-scoped playback (wake-word detection is not implemented)
 Text Generation LLM inference, 100% on-device |
 Voice AI STT → LLM → TTS pipeline, fully offline | |
 Image Generation On-device diffusion model |
 Visual Language Model Vision + language understanding on-device | |
import RunAnywhere
import LlamaCPPRuntime
// 1. Initialize
LlamaCPP.register()
try RunAnywhere.initialize()
// 2. Load a model
var load = RAModelLoadRequest()
load.modelID = "smollm2-360m"
load.category = .language
load.framework = .llamaCpp
_ = await RunAnywhere.loadModel(load)
// 3. Generate
var req = RALLMGenerateRequest()
req.prompt = "What is the capital of France?"
let result = try await RunAnywhere.generate(req)
print(result.text) // "Paris is the capital of France."
Add the MLX backend (import RunAnywhereMLX; MLX.register()) for Apple-native LLM, VLM, STT, TTS, and embeddings on Apple silicon.
Install via Swift Package Manager:
https://github.com/RunanywhereAI/runanywhere-sdks
import ai.runanywhere.proto.v1.ModelCategory
import ai.runanywhere.proto.v1.SDKEnvironment
import com.runanywhere.sdk.llm.llamacpp.LlamaCPP
import com.runanywhere.sdk.public.RunAnywhere
import com.runanywhere.sdk.public.extensions.*
import com.runanywhere.sdk.public.types.RAModelInfo
import com.runanywhere.sdk.public.types.RAModelLoadRequest
// 1. Initialize (in a coroutine scope)
LlamaCPP.register()
RunAnywhere.initialize(
context = this,
environment = SDKEnvironment.SDK_ENVIRONMENT_DEVELOPMENT,
)
// 2. Download and load a model
val modelId = "smollm2-360m-instruct-q8_0"
RunAnywhere.downloadModelStream(RAModelInfo(id = modelId)).collect { /* progress */ }
RunAnywhere.loadModel(
RAModelLoadRequest(model_id = modelId, category = ModelCategory.MODEL_CATEGORY_LANGUAGE),
)
// 3. Generate
val result = RunAnywhere.generate("What is the capital of France?")
println(result.text) // "Paris is the capital of France."
Install via Gradle (Maven Central):
dependencies {
implementation("io.github.sanchitmonga22:runanywhere-sdk:0.20.11")
implementation("io.github.sanchitmonga22:runanywhere-llamacpp:0.20.11")
// Optional: STT / TTS / VAD
// implementation("io.github.sanchitmonga22:runanywhere-onnx:0.20.11")
}
import 'package:runanywhere/runanywhere.dart';
import 'package:runanywhere_llamacpp/runanywhere_llamacpp.dart';
// 1. Initialize
LlamaCpp.register();
await RunAnywhere.initialize();
// 2. Download and load a model
await RunAnywhere.downloadModel('smollm2-360m');
await RunAnywhere.llm.load('smollm2-360m');
// 3. Generate
final response = await RunAnywhere.llm.chat('What is the capital of France?');
print(response); // "Paris is the capital of France."
Install via pub.dev:
dependencies:
runanywhere: ^0.20.11
runanywhere_llamacpp: ^0.20.11 # LLM/VLM text generation
# runanywhere_onnx: ^0.20.11 # STT, TTS, VAD, voice agent
# runanywhere_mlx: ^0.20.11 # Apple-native LLM/VLM/STT/TTS/embeddings
# runanywhere_qhexrt: ^0.20.11 # Snapdragon Hexagon NPU
import { RunAnywhere, SDKEnvironment } from '@runanywhere/core';
import { LlamaCPP } from '@runanywhere/llamacpp';
// 1. Initialize
await RunAnywhere.initialize({ environment: SDKEnvironment.SDK_ENVIRONMENT_DEVELOPMENT });
LlamaCPP.register();
// 2. Download and load a model
await RunAnywhere.downloadModel('smollm2-360m');
await RunAnywhere.loadModel('smollm2-360m');
// 3. Generate
const result = await RunAnywhere.generate('What is the capital of France?');
console.log(result.text); // "Paris is the capital of France."
Install via npm:
npm install @runanywhere/core@0.20.11 @runanywhere/llamacpp@0.20.11
# optional backends: @runanywhere/onnx @runanywhere/mlx @runanywhere/qhexrt
import { RunAnywhere, SDKEnvironment } from '@runanywhere/web';
import { LlamaCPP } from '@runanywhere/web-llamacpp';
// 1. Initialize
await RunAnywhere.initialize({ environment: SDKEnvironment.SDK_ENVIRONMENT_DEVELOPMENT });
await LlamaCPP.register({ acceleration: 'auto' }); // WebGPU when available, WASM otherwise
await RunAnywhere.completeServicesInitialization();
// 2. Load a model
await RunAnywhere.loadModel({ modelId: 'qwen2.5-0.5b' });
// 3. Generate
const result = await RunAnywhere.generate({
prompt: 'What is the capital of France?',
});
console.log(result.text); // "Paris is the capital of France."
Install via npm:
npm install @runanywhere/web@0.20.11 @runanywhere/web-llamacpp@0.20.11
# @runanywhere/web-onnx for STT/TTS/VAD/embeddings in the browser
const { RunAnywhere } = require('@runanywhere/electron');
// 1. Initialize
RunAnywhere.initialize();
// 2. Load a model (catalog id or a local path)
const llm = await RunAnywhere.loadLLM('qwen2.5-0.5b');
// 3. Generate (streaming)
for await (const token of llm.generate('What is the capital of France?')) {
process.stdout.write(token);
}
llm.unload();
RunAnywhere.shutdown();
A native N-API addon over the C core. Inference runs in an isolated Electron utility process and streams to the renderer over a MessagePort. LLM, VLM, STT, TTS, embeddings, RAG, structured output, tool calling, and a voice pipeline, with a prebuilt win32-x64 addon. CUDA is available as an opt-in source build.
Install: build from source (Windows x64 preview), see the SDK README for steps.
import runanywhere as ra
from runanywhere import LlmOptions
# 1. One call brings the SDK up
ra.initialize()
# 2. Stream tokens (the model auto-downloads and auto-loads)
for event in ra.llm.generate_stream("What is the capital of France?",
LlmOptions(model="qwen2.5-0.5b")):
if event.is_token:
print(event.text, end="", flush=True)
# 2b. Or async
# async for event in ra.llm.agenerate_stream("..."):
# ...
# 3. Or grab the whole result, metrics included
result = ra.llm.generate("Capital of France? One word.")
print(result.text, result.tokens_per_second) # "Paris" 41.2
Namespaces per modality (llm, vlm, stt, tts, vad, embeddings, rag, models), an a-prefixed async twin for every blocking verb, structured output and tool calling, with prebuilt wheels that bundle the native runtime. CUDA is available as an opt-in source build.
Install via pip:
pip install runanywhere==0.20.11
$ rcli pull qwen3
pulling qwen3-0.6b ▕████████████▏ 100% 639 MB/639 MB 32 MB/s
$ rcli run qwen3 "Reply with exactly: RCLI WORKS" --no-think
RCLI WORKS
$ rcli tts --text "RunAnywhere runs models on device." --output hello.wav
$ rcli stt --input hello.wav
Run anywhere runs models on device.
$ rcli voice --input question.wav --output reply.wav # full STT > LLM > TTS turn
$ rcli serve qwen3 # OpenAI-compatible API on :8080
Also: rcli run --image photo.jpg (VLM), rcli vad, rcli embed, rcli image (diffusion, Apple), rcli lora, and --json on everything.
Install (macOS Apple Silicon, Linux x86_64/aarch64, Windows x86_64):
brew install runanywhereai/tap/rcli
# or
curl -fsSL https://raw.githubusercontent.com/RunanywhereAI/runanywhere-sdks/main/rcli/scripts/install.sh | sh
| SDK | Platforms | Status | Install | Docs |
|---|
| Swift | iOS 17.5+, macOS 14.5+ | Stable | Swift Package Manager | docs.runanywhere.ai/swift |
| Kotlin | Android API 24+ | Stable | Gradle (io.github.sanchitmonga22:runanywhere-sdk) | docs.runanywhere.ai/kotlin |
| Flutter | iOS, Android | Beta | pub.dev (runanywhere) | docs.runanywhere.ai/flutter |
| React Native | iOS, Android | Beta | npm (@runanywhere/core) | docs.runanywhere.ai/react-native |
| Web | Chromium, Safari, Firefox | Beta | npm (@runanywhere/web) | SDK README |
| Electron | Windows x64 desktop | Preview | Build from source | SDK README |
| Python | Windows, macOS, Linux | Alpha | pip (runanywhere) | SDK README |
| rcli | macOS, Linux, Windows | Stable | Homebrew / install script | CLI README |
| Feature | Swift | Kotlin | Flutter | RN | Web | Electron | Python | rcli |
|---|
| LLM generation + streaming | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Vision language models (VLM) | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Computer-use agent (CUA) | Yes | Yes | Yes | Yes | API only | n/a | n/a | n/a |
| Speech-to-Text | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Text-to-Speech | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Voice activity detection | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Voice agent pipeline | Yes | Yes | Yes | Yes | Yes | Yes | Stub | Yes |
| Wake word | No | No | No | No | No | No | No | No |
| Embeddings | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| RAG (with streaming) | Yes | Yes | Yes | Yes | Yes* | Yes | Yes | n/a |
| Structured output (JSON) | Yes | Yes | Yes | Yes | Yes | Yes | Yes | n/a |
| Tool calling | Yes | Yes | Yes | Yes | Yes | Yes | Yes | n/a |
| Image generation (diffusion) | Yes | Yes | Yes | Yes | n/a | n/a | Stub | Yes |
| LoRA adapters | Yes | Yes | Yes | Yes | Yes | Partial | Stub | Yes |
| Diarization (standalone) | Yes | Yes | Gated | Yes | Yes | n/a | Stub | n/a |
| Segmentation | Yes | Yes | Gated | Yes | Yes | n/a | Stub | n/a |
capabilities() discovery | Yes | Yes | Yes | Yes | Yes | Partial | Yes | n/a |
| iOS, iPadOS (Swift), Android (Kotlin) |
| React Native, Flutter, Web, Electron |
_runanywhere-connect._tcpidl/connect.proto, rac_connect_*)RunAnywhereMain.connect() is local MessagePort / utility-process IPC inside one Electron app. It is unrelated to LAN Connect| Engine | Modalities | Runs on | Notes |
|---|
| QHexRT | LLM, VLM, STT, TTS, embeddings, inpainting | Snapdragon Hexagon NPU (v75 / v79 / v81) | RunAnywhere's own NPU runtime, details below |
| MLX | LLM, VLM, STT, TTS, embeddings | Apple silicon | Apple-native inference via mlx-swift, safetensors models |
| llama.cpp | LLM, VLM | Everywhere: Metal on Apple, CUDA opt-in on Windows/Linux, WebGPU + WASM in the browser, CPU with NEON/AVX | GGUF models |
| sherpa + ONNX | STT, TTS, VAD, embeddings | All platforms | sherpa-onnx for speech, ONNX Runtime for embeddings and RAG |
| Core ML | Image generation (diffusion) | iOS, macOS | Core ML dispatches each layer across CPU, GPU, and the Apple Neural Engine |
| Platform | Apple Foundation Models, system TTS | iOS, macOS, Android | Native OS capabilities behind the same API |
| Cloud | Hybrid STT | All platforms | Optional confidence-cascade routing to hosted providers |
| Model | Task | Params | Decode | Time to first token |
|---|
| LFM2.5-230M | LLM | 0.23 B | 164 tok/s | 32 ms |
| Qwen3-0.6B | LLM | 0.6 B | 33 tok/s (prefill up to 3,692 tok/s) | 127 ms |
| Llama-3.2-1B | LLM | 1.2 B | 16.3 tok/s | 56 ms |
| Phi-tiny-MoE | MoE LLM | 3.8 B (1.1 B active) | 5-7 tok/s | ~2.5 s |
| InternVL3.5-1B | VLM | 1 B | 37 tok/s | 290 ms |
| Whisper base | ASR | 74 M | ~5x real-time | n/a |
| MeloTTS-EN | TTS | n/a | ~4.5x real-time | n/a |
| Model | Task | Params | Bundle |
|---|
| Llama-3.2-1B | LLM | 1.2 B | llama3_2_1b_HNPU |
| LFM2.5-230M / 350M | LLM | 0.23 / 0.35 B | lfm2_5_230m_HNPU · lfm2_5_350m_HNPU |
| LFM2.5-2.6B | LLM | 2.6 B | lfm2_5_2_6b_HNPU |
| Qwen3.5-0.8B / 2B / 4B | LLM | 0.8-4 B | qwen3_5_0_8b_HNPU · 2b · 4b |
| Bonsai 1-bit family | LLM | 1.7 / 4 / 8 / 27 B | 1-bit and ternary builds; Bonsai-27B runs on Hexagon v81 |
| Gemma-4-E2B / E4B | LLM + VLM | ~2 / 4 B | gemma4_e2b_HNPU · gemma4_e4b_HNPU |
| Phi-tiny-MoE | MoE LLM | 3.8 B | phi_tiny_moe_HNPU |
| DeepSeek-R1-Distill-Qwen | LLM | 1.5 / 7 B | 1.5b · 7b |
| Cosmos3-Edge | LLM | edge | NVIDIA model family, Hexagon v79 |
| Qwen3-VL-2B | VLM | 2 B | qwen3_vl_HNPU |
| InternVL3.5-1B | VLM | 1 B | internvl3_5_1b_HNPU |
| Whisper base / small | ASR | 74 / 244 M | whisper_base_HNPU · whisper_small_HNPU |
| Moonshine tiny / base | ASR | n/a | moonshine_base_HNPU |
| MeloTTS-EN | TTS | n/a | melotts_en_HNPU |
| Magpie-TTS Multilingual | TTS | 357 M | magpie_tts_357m_HNPU |
| Kitten TTS mini / micro | TTS | n/a | Hexagon v75 |
| EmbeddingGemma-300M | Embeddings | 300 M | embeddinggemma_300m_HNPU |
| Type | Models | Engine |
|---|
| LLM | SmolLM2, Qwen 3 / 2.5, Llama 3.2, LFM2, Mistral 7B (GGUF) | llama.cpp |
| LLM / VLM (Apple) | Qwen3, SmolVLM2, and other mlx-community safetensors models | MLX |
| VLM | SmolVLM2, LFM2-VL, Qwen2-VL (GGUF + mmproj) | llama.cpp |
| Speech-to-Text | Whisper Tiny / Base, Moonshine | sherpa + ONNX |
| Text-to-Speech | Piper voices, Kokoro, Kitten TTS | sherpa + ONNX |
| VAD | Silero VAD | sherpa + ONNX |
| Embeddings | MiniLM, EmbeddingGemma | ONNX Runtime |
| Image generation | Stable Diffusion | Core ML |
| Platform | Source | Get it |
|---|
| iOS | RunanywhereAI/runanywhere-ios | App Store |
| Android | RunanywhereAI/runanywhere-android | Google Play |
| Web | RunanywhereAI/runanywhere-web | Build from source |
| Electron | RunanywhereAI/runanywhere-electron | Build from source (Windows) |
| React Native | bindings/react-native/example | Build from source |
| Flutter | bindings/flutter/example | Build from source |
| SDK | Harness | Run it |
|---|
| Swift | bindings/swift/example | ./run example ios run |
| Kotlin | bindings/kotlin/example | ./run example android install |
| Web | bindings/web/example | ./run example web dev |
runanywhere-sdks/
├── core/ # Shared C/C++ core behind a C ABI — all business logic
│
├── bindings/ # Thin language bindings over core/
│ ├── swift/ # iOS/macOS SDK (XCFramework) + example/
│ ├── kotlin/ # Android SDK (JNI) + example/
│ ├── flutter/ # Flutter SDK (Dart FFI) + example/
│ ├── react-native/ # React Native SDK (Nitro/JSI) + example/
│ ├── web/ # Web SDK (WebAssembly / WebGPU) + example/
│ ├── electron/ # Electron SDK (N-API addon) + example/
│ ├── python/ # Python SDK (pybind11) + example/
│ ├── proto-ts/ # @runanywhere/proto-ts, the IDL's TypeScript binding
│ └── shared-apple/ # Apple transport shared by the RN + Flutter bindings
│ # (iOS/Android/Web/Electron consumer apps live in their own repos)
│
├── rcli/ # rcli, the terminal app built on core/
├── engines/ # llamacpp, mlx, sherpa, onnx, neurt, qhexrt, cloud
├── runtimes/ # cpu, coreml, onnxrt compute adapters
├── idl/ # Protobuf schemas, generated bindings per language
└── docs/ # Documentation
| Platform | Minimum |
|---|
| iOS | 17.5+ |
| macOS | 14.5+ |
| Android | API 24 (7.0), arm64 recommended |
| Web | Chrome 96+ / Edge 96+, Chrome 120+ for WebGPU |
| React Native | 0.83.1+, 0.85+ recommended (Node.js 22.12+) |
| Flutter | 3.44+ (Dart 3.12+) |
| Electron | Windows x64 (preview) |
| Python | 3.9+ on Windows, macOS, Linux (3.12+ recommended) |
| rcli | macOS arm64, Linux x86_64 / aarch64, Windows x86_64 |
Surfaced from shared tags and platforms — no rankings paid for.