cactus

1.4.3-betaindexed

Fast, lightweight inference framework for energy-efficient on-device AI: numerical computation graph API, OpenAI-compatible inference engine, INT8 optimizations and model/tooling for compact, low-power deployments.

AndroidNative·cactus-compute/cactus

5.4k

Stars

Used by

dependents

Health

/ 100

Cactus

A hybrid edge-cloud AI engine for mobile devices & wearables.

┌─────────────────┐
│  Cactus Engine  │ ←── OpenAI-compatible APIs for text, speech, and vision.
└─────────────────┘     
         │
┌─────────────────┐
│  Cactus Graph   │ ←── Zero-copy computation graph
└─────────────────┘     
         │
┌─────────────────┐
│ Cactus Kernels  │ ←── CPU/GPU kernels for (Apple, Samsung, Pixel, etc.)
└─────────────────┘     
         │
┌─────────────────┐
│ Cactus Quants   │ ←── Custom rotation-based quantization technique
└─────────────────┘

Quick Demo (Mac)

Step 1: brew install cactus-compute/cactus/cactus
Step 2: cactus run

Cactus Engine

Example response from Gemma4-E2B

Cactus Graph

Inference Speed

LLM: Gemma-4-E2B-CQ4 (1k-context prefill / decode for 100 tokens)
VLM: Gemma-4-E2B-CQ4 (256px image encode time / decode)
Transcribe: Parakeet-TDT-0.6B-CQ4 (20s audio end-to-end transcribe time)
1k-Context RAM: peak MB during the LLM benchmark
No speculative decode or MTP, pure decode

Command: cactus benchmark [optional --ios or --android]

N/B: With 1k-context prefill and decode for 100 runs on M5 Max

LFM2.5-VL-1.6B = 289toks/sec
Qwen3-1.7B = 155toks/sec
LFM2.5-VL-450m = 472toks/sec, image encodes in 43ms
LFM22.5-VL-230m = 555toks/sec

Output Quality

Gemma-4-E2B-it accuracy across bit widths, averaged over 3 seeds.
CQ3.26 and CQ2.54 are mixed-precision, CQ2/CQ3/CQ4 are uniformly quantized.
Full results in docs/cactus_quants.md:

Supported Models

Any HuggingFace model can be converted using cactus convert [HF-Name], though experimental.
Liquid, Gemma. whisper. parakeet and Qwen model families are especially tested.
Some models have been pre-uploaded here, just run cactus download [HF-Name].
cactus run [HF-Name] albeit first downloads or convert the model if not found.

Needle

Needle is a 26m parameter model for on-device tool calling:

cactus run Cactus-Compute/needle [--tools my_tools.json]  # OpenAI function-calling format; demo toolset by default

Learn More

Bindings

Using this repo

Citation

If you use Cactus in your research, please cite it as follows:

@software{cactus,
  title        = {Cactus: AI Inference Engine for Phones & Wearables},
  author       = {Ndubuaku, Henry and Cactus Team},
  url          = {https://github.com/cactus-compute/cactus},
  year         = {2025}
}

N/B: Scroll all the way up and click the shields link for resources!

Related libraries

Surfaced from shared tags and platforms — no rankings paid for.

koog★ 4.4k

JetBrainsFramework designed for building AI agents with tool interaction, complex workflows, semantic search, and persistent memory. Offers modular architecture, real-time processing, and comprehensive tracing.Shared: llm, ai

Llamatik★ 161

ferranponsOn-device and remote LLM inference via native llama.cpp bindings, offering embeddings, context-aware text generation (streaming & non-streaming), lightweight HTTP client/server and GGUF model support.Shared: llm, ai

deepseek-kotlin★ 8

OremifProvides seamless access to a REST API for integrating chat functionalities, enabling configurable client setup, request handling, and streaming of response chunks for efficient data processing.Shared: llm, ai

llm-typewriter★ 6

NadeemIqbalStreaming-text typewriter for LLM apps: renders streaming tokens with live progressive Markdown, per-language syntax-highlighted code blocks, human-like speed curves, configurable cursor, tap-to-skip, and accessibility support.Shared: llm, ai

prompt-bar★ 6

NadeemIqbalAI chat composer UI with multi-line auto-growing input, slash-command autocomplete, @mention dropdown, attachment chips/previews, unified Send/Sending/Stop state, voice support, templates and live token counter.Shared: llm, ai

deviceai★ 5

deviceai-labsOn-device AI runtime enabling speech recognition, TTS, and local LLM inference with offline RAG, auto model downloads, streaming generation, and GPU acceleration for low-latency, privacy-preserving apps.Shared: llm, ai

Cactus

A hybrid edge-cloud AI engine for mobile devices & wearables.

┌─────────────────┐ │ Cactus Engine │ ←── OpenAI-compatible APIs for text, speech, and vision. └─────────────────┘ │ ┌─────────────────┐ │ Cactus Graph │ ←── Zero-copy computation graph └─────────────────┘ │ ┌─────────────────┐ │ Cactus Kernels │ ←── CPU/GPU kernels for (Apple, Samsung, Pixel, etc.) └─────────────────┘ │ ┌─────────────────┐ │ Cactus Quants │ ←── Custom rotation-based quantization technique └─────────────────┘

Inference Speed

LLM: Gemma-4-E2B-CQ4 (1k-context prefill / decode for 100 tokens)

VLM: Gemma-4-E2B-CQ4 (256px image encode time / decode)

Transcribe: Parakeet-TDT-0.6B-CQ4 (20s audio end-to-end transcribe time)

1k-Context RAM: peak MB during the LLM benchmark

No speculative decode or MTP, pure decode

Command: cactus benchmark [optional --ios or --android]

N/B: With 1k-context prefill and decode for 100 runs on M5 Max

LFM2.5-VL-1.6B = 289toks/sec

Qwen3-1.7B = 155toks/sec

LFM2.5-VL-450m = 472toks/sec, image encodes in 43ms

LFM22.5-VL-230m = 555toks/sec

Supported Models

Any HuggingFace model can be converted using cactus convert [HF-Name], though experimental.

Liquid, Gemma. whisper. parakeet and Qwen model families are especially tested.

Some models have been pre-uploaded here, just run cactus download [HF-Name].

cactus run [HF-Name] albeit first downloads or convert the model if not found.

Citation

If you use Cactus in your research, please cite it as follows:

@software{cactus, title = {Cactus: AI Inference Engine for Phones & Wearables}, author = {Ndubuaku, Henry and Cactus Team}, url = {https://github.com/cactus-compute/cactus}, year = {2025} }

N/B: Scroll all the way up and click the shields link for resources!

Mac M5 Max	2964tps / 154tps	0.09s / 168tps	0.15s	1348MB
Mac M4 Pro	1963tps / 101tps	0.25s / 112tps	0.21s	1225MB
Mac M3 Pro	1294tps / 64tps	0.40s / 72tps	0.37s	735MB
iPad/Vision Pro M5	1336tps / 71tps	0.25s / 80tps	0.27s	703MB
iPhone 17 Pro	729tps / 37tps	0.5s / 39tps	0.51s	644MB
iPhone 15 Pro	517tps / 26tps	1.15s / 27tps	0.82s	633MB

ARC-E	73.80	73.73	74.20	68.20	50.80
ARC-C	56.47	52.47	51.53	37.20	24.73
HellaSwag	46.93	47.07	45.20	40.73	35.87
WinoGrande	61.00	61.13	59.60	60.13	51.27
MMLU	62.33	59.45	57.63	47.19	33.18
GPQA	34.34	34.34	31.82	30.81	23.23
GSM8K	73.67	71.20	66.20	22.00	0.40
HumanEval	54.88	57.11	53.66	15.24	1.02
BFCL Simple	92.00	92.42	91.50	82.25	18.75
BFCL Multi	89.00	88.33	89.00	52.50	13.67
BFCL Parallel	84.00	83.67	82.50	30.00	3.33
BFCL Parallel-Multi	78.00	83.33	82.00	37.00	1.33

Cactus Engine	C	Chat completion, streaming, tool calling, transcription, embeddings, RAG, vision, vector index, cloud handoff
Cactus Graph	C++	Tensor operations, matrix multiplication, attention, normalization, activation functions
Cactus Kernels	C++	ARM NEON SIMD kernels for matmul, attention, convolution, quantization, DSP, image processing
Cactus Quants	C++	Rotation-and-codebook quantization from 4-bit to 1-bit for all weight tensors
Cactus Hybrid	C/Python	Route hard queries to the cloud automatically based on local model confidence
Python Package	Python	Python package and CLI