llama.kt
llama.cpp for Android, as a Kotlin library. CPU, Vulkan and OpenCL in one native library. CPU and Vulkan are validated on a Mali-G68. OpenCL is compiled in and not yet tested on Adreno hardware. Apache 2.0.
What it does
- Loads any GGUFReads the header in milliseconds for quant, size and context before the full load.
- StreamingTokens as a Kotlin Flow, UTF-8 safe, stops on EOS or a hard cap.
- Chat templates from the modelThe template comes from the GGUF itself, so any instruct model works as is.
- KV cache sessionsFollow-ups reuse the cached history. Sessions save and restore across restarts.
- Reasoning splitReplies come split into answer and reasoning by llama.cpp's own parser for that template. Best effort, keep a fallback.
git submodule add https://github.com/tensai-app/llama.kt libs/llama.ktgit -C libs/llama.kt submodule update --init --recursive# vendors the llama.cpp sources, required before the first buildbash libs/llama.kt/scripts/bootstrap.sh// settings.gradle.ktsinclude(":llama-kt")project(":llama-kt").projectDir = file("libs/llama.kt/llama-kt") // app/build.gradle.ktsdependencies { implementation(project(":llama-kt"))}import com.tensai.llamakt.* val engine = LlamaEngine()engine.load(path = "/path/to/model.gguf", nGpuLayers = 0, nCtx = 4096) val messages = listOf(ChatMessage("user", "Who are you?"))engine.chat(messages).collect { token -> print(token) } // streams, stops on EOS engine.free()The AAR is not published yet. The git submodule is the supported path for now. Maven Central may come later.
Measured
OPPO CPH2251 · Dimensity 900 (2× A78 2.4 GHz + 6× A55 2.0 GHz, Mali-G68 MC4, LPDDR4X) · Android 13 · nCtx 4096 · temp 0 · ~20-token prompt · CPU = 2 big-core threads · llama.cpp b9769 · tok/s
| Model | Quant | Backend | pp | tg |
|---|---|---|---|---|
| Qwen2.5-0.5B-Instruct | Q4_K_M | CPU(faster backend) | 29.5 | 15.0 |
| Vulkan | 7.0 | 9.0 | ||
| Gemma-3-1B-it | Q4_K_M | CPU(faster backend) | 10.9 | 6.0 |
| Vulkan | 3.7 | 5.4 | ||
| Qwen3-1.7B | Q4_K_M | CPU(faster backend) | 8.6 | 5.9 |
| Vulkan | 2.1 | 4.2 | ||
| Llama-3.2-3B-Instruct | Q4_K_M | CPU(faster backend) | 6.0 | 3.9 |
| Vulkan | 1.2 | 2.9 | ||
| Phi-4-mini (3.8B) | Q4_K_M | CPU | 3.9 | 1.7 |
| Vulkan(faster backend) | 0.8 | 2.3 | ||
| Gemma-3n-E2B-it | Q4_K_M | CPU | 1.9 | 1.1 |
| Vulkan(faster backend) | 1.8 | 1.9 | ||
| Qwen2.5-7B-Instruct | Q4_K_M | CPU | 1.4 | 0.2 |
A selection of the models measured, in tok/s. pp = prompt processing, tg = generation. The dot marks the faster backend for each model. The full table, with more models and backends (Gemma 4 included), is in the README.
1.7 to 2.4 s vs ~48 s
Time to first token over 4 dependent turns, history 0 to 529 tokens, with KV prefix reuse vs a full re-prefill.
10.7 s to 82 ms
Time to first token after a process kill, 171-token prompt, restoring a saved session.
Upstream
llama.cpp and llama.rn are pinned. Fixes go upstream where they can; the rest live as patches applied at bootstrap.
- Vulkan UMA fixOverflow-safe CEIL_DIV in ggml_vk_matmul for mobile drivers that report UINT32_MAX work groups. Fixed batched matmuls aborting at load.merged#25245
- 0002-abort-callback-mid-graph-cancelAbort callback checked per graph node, so cancelling a long prefill does not wait for the whole batch.local
- 0003-vulkan-device-dispatcher-initLoads device-level Vulkan function pointers after createDevice, and turns off buffer_device_address on Mali when it is not exposed.local
- 0004-stale-embd-kv-invariantNever trusts KV cells that were not decoded: position check, SWA window guard, trimmed on abort.local
- 0005-load-stage-callbackLoad stages (weights, context, warm-up) reported through the progress callback.local
- 0006-kv-reuse-past-divergenceKeeps the KV cache up to the point where a new prompt diverges from the cached one, instead of dropping it.local
- 0007-vulkan-submit-mflops-envMakes the amount of work per Vulkan submit tunable through an environment variable.local
- 0008-vulkan-queue-prioritySets an explicit priority on the Vulkan compute queue.local
- 0009-vulkan-matmul-splitOptionally splits large Vulkan matmuls along M into smaller dispatches for mobile drivers.local