Does it really work on a phone?

Yes. Quantized GGUF models run in a modern phone's unified memory, and on Apple silicon they run via MLX. Speed and maximum context depend on model size and device memory.

Isn't the quality worse than a big cloud model?

It is not in the same tier as top cloud models. POCKET targets a practical, on-device 35B-class, and its edge is matching a leading on-device model on quality while beating it on speed.

What does 'slow prefill' mean?

The initial step of reading a long prompt is comparatively slow. It is minor for short and medium chats but noticeable when you paste thousands of tokens at once.

VIDRAFT / Insights / On-device AI

On-device AI

Can you run a large LLM without a GPU?

On-device AI runs a 35B-class open model on a CPU, phone, or laptop — no GPU. VIDRAFT POCKET decodes 2.69x faster than the most-downloaded on-device model at matched quality.

Published 2026-07-24About 3min readby VIDRAFT

Quick answer

Yes. Compressing a large language model with 4-bit quantization and mixture-of-experts (MoE) lets it run on the CPU of a phone, laptop, or air-gapped server — no GPU. VIDRAFT POCKET runs a 35B-class open model this way and, at matched quality, decodes 2.69x faster than the most-downloaded on-device model.

Can you really run an LLM without a GPU?

You can. Quantizing weights to around 4 bits and using mixture-of-experts (MoE) to cut active parameters sharply lowers the compute and memory needed, so a plain CPU with unified memory can run inference. llama.cpp (GGUF) and Apple's MLX are the common runtimes.

LLMs demand GPUs because they multiply billions of weights at high precision. But at inference, lower precision costs little quality, and MoE activates only some experts per token, so real compute is far below the total parameter count. Combined, a 26-35B-class model shrinks to 10-20GB and fits in a modern laptop's memory.

What does on-device AI actually buy you?

Privacy, offline operation, cost, and sovereignty. Data never leaves the device, so it is strong for security and regulation; it works with no network; and it removes both cloud inference bills and GPU procurement. It is decisive exactly where cloud AI cannot go — closed networks, defense, healthcare.

Cloud LLMs send every input to an external server. On-device inference keeps input and output inside the device, a fundamental advantage for confidential work, and it runs where there is no internet. That is the precondition for sovereign AI in public and regulated domains that forbid data export.

What are POCKET's measured numbers?

At a matched quality target it measured 2.69x faster CPU decoding and 2.22x faster decoding on a single consumer GPU. Answer quality was on par (matched) on standard evals. On very long prompts (e.g. 8k) prefill is slower at 0.41x, and we publish that losing axis openly.

POCKET is built on Google's open Gemma model, tuned for Korean and shipped as GGUF and MLX. The point is a wide lead in the token generation (decode) speed users feel, while holding quality. It is strong for chat, agents, and short-to-medium prompts, with a trade-off on long-document batch prefill.

Where should you use it?

A phone in your hand, a GPU-less laptop, and an air-gapped server. Defense, healthcare, and administration — where data cannot leave the building and cloud AI cannot enter — are the core market for on-device, sovereign AI.

On-device models extend to personal-device assistants, offline field tools, and the internal AI of tightly regulated organizations. They put modern language-model capability directly where the data lives, with no GPU infrastructure.

Frequently asked questions

Does it really work on a phone?: Yes. Quantized GGUF models run in a modern phone's unified memory, and on Apple silicon they run via MLX. Speed and maximum context depend on model size and device memory.
Isn't the quality worse than a big cloud model?: It is not in the same tier as top cloud models. POCKET targets a practical, on-device 35B-class, and its edge is matching a leading on-device model on quality while beating it on speed.
What does 'slow prefill' mean?: The initial step of reading a long prompt is comparatively slow. It is minor for short and medium chats but noticeable when you paste thousands of tokens at once.

Sources

LLM engineering

What is model quantization — is a 4-bit LLM still smart?

LLM engineering

Can you make an AI model smarter without training? Model merging

Physical AI

Can you teach a robot Korean without touching its firmware?

↖ Home — vidraft.net

This article is based on VIDRAFT's public, measured data and external sources. Performance figures are measurements under the stated conditions and may vary by environment.