Can you run a 35B AI model with no GPU?
How VIDRAFT's POCKET Box and POCKET-35B run a large language model on-device, CPU-only, and how fast it is.
Yes. VIDRAFT's POCKET Box is a CPU-only appliance with no GPU that runs POCKET-35B on-device, a 34.66B-parameter Sparse MoE model that activates only ~3B parameters per token. On a 16-thread Xeon CPU it reached 27.0 tok/s at IQ1_M (developer-measured under stated conditions, not third-party verified).
Can you run a large language model without a GPU?
POCKET-35B is a Sparse MoE model with 34.66B total parameters that activates only ~3B per token, built on Darwin-36B-Opus (a Qwen3.5-family MoE). In-domain expert pruning cuts 256 experts down to 128, halving the model size. It runs on unmodified upstream llama.cpp and MLX with no runtime fork required.
Why does POCKET-35B run faster on CPU?
Because it reads fewer weights per token, the weaker the hardware, the more the MoE wins. On a 16-thread Xeon CPU, POCKET-35B hit 27.0 tok/s at IQ1_M, 2.69x faster than the same-size 1-bit Bonsai-27B (10.1 tok/s). MoE-aware mixed-precision quantization compresses Korean harder and prunes English harder.
How fast is POCKET Box on different hardware?
On H100 it ran 197 vs 89 tok/s, a 2.22x edge over the same-size baseline. On mobile, iPhone's POCKET-EN ships as a 5.3GB mixed-precision GGUF and Android's POCKET-KR as a 5.1GB MLX 2-bit build, with 88% and 94% routing consistency respectively. All figures are self-measured under stated conditions and are not third-party verified.
How do you install POCKET-35B and which quantization should you use?
Because POCKET-35B runs on unmodified llama.cpp and MLX, you can use the standard toolchain directly. Extreme low-bit quantization such as IQ1_M is fast but degrades Korean output, so Q4+ is recommended for real use. The benchmark numbers are developer self-measured and provided for reference.
Frequently asked questions
- Does POCKET Box need a GPU?
- No. POCKET Box is a CPU-only appliance with no GPU at all.
- Can it run on iPhone or Android?
- Yes. iPhone's POCKET-EN is a 5.3GB GGUF and Android's POCKET-KR is a 5.1GB MLX 2-bit build.
- Which quantization is recommended?
- Q4 or higher is recommended for Korean quality; extreme low-bit like IQ1 is faster but degrades Korean output.
Related
This article is based on VIDRAFT public, measured data and external sources. Performance figures are measurements under the stated conditions and may vary by environment.