PortfolioAlpaca / Inference engineSoftware engineeringIndex / 01

01/Inference engine/2026

Alpaca

Native LLM inference for Apple Silicon

A Swift-first inference runtime with custom Metal kernels, GGUF loading and on-device generation. No llama.cpp, MLX, Core ML, ONNX or Python at runtime.

Role
Design and engineering
Stack
Swift 6, Metal
Platform
macOS 14+, iOS 17+
Status
0.1.0 pre-release
0200400600161285121,0242,0484,096549513469498412378109DECODE, TOKENS PER SECONDCONTEXT, TOKENSALPACALLAMA.CPPALPACA, BEFOREQ8_0 / SMOLLM2-135M / APPLE M5 / MEDIAN OF 7
Fig. 01.1 / Decode throughput against context length, measured

Every kernel,
written here.

Alpaca runs a Llama-style language model on Apple Silicon with nothing underneath it. The tensor engine, GGUF parser, tokenizer, forward pass, sampler and every Metal kernel are written in the repository. Python and PyTorch appear only offline, to generate reference fixtures.

Correctness came first. Each operation has a deterministic CPU reference, and the Metal path is tested against it, and against PyTorch, Hugging Face and the gguf package, before any speed is claimed.

let model = try await LanguageModel.load(from: modelURL)

let stream = model.generate(
    prompt: prompt,
    configuration: .init(maxTokens: 256, temperature: 0.7, seed: 42)
)
for try await token in stream { print(token.text, terminator: "") }

01/Five decisions

  1. 01

    Weights are never copied

    The whole memory-mapped GGUF file is wrapped as one no-copy Metal buffer in unified memory, so resident weight memory is about the file size and nothing is duplicated.

  2. 02

    Tiled prefill attention

    A FlashAttention-style kernel on simdgroup matrices, with online softmax and causal block skipping. It took attention for a 2,048-token prompt from 1,111 ms to about 130 ms.

  3. 03

    Split-K decode attention

    One threadgroup per key-value head and position split, so each key and value is read once for the three query heads that share it. Decode at 2,048 tokens of context went from 157 to 310 tokens per second.

  4. 04

    Decoding stays on the GPU

    An arg-max kernel feeds the next step directly, with three command buffers in flight, so the CPU only streams tokens out. Q8_0 decode went from 437 to 521 tokens per second at short context.

  5. 05

    Model files are untrusted input

    Every length in the GGUF parser is bounded by the bytes actually present and all arithmetic is overflow-checked. It survived 20,000 mutation-fuzzed files without a crash.

02/Evidence

  • 549 tok/sDecode at 16 tokens of context, Q8_0. llama.cpp: 430
  • 1.28×Decode speed against llama.cpp on the same files, in the same run
  • 17,351Tokens per second prefilling 128 tokens, at parity with llama.cpp (17,163)
  • 72/72Greedy tokens identical to the reference, on every weight format

SmolLM2-135M-Instruct on an Apple M5 (10-core GPU, 16 GB), macOS 27.0, Swift 6.4 release build. Median of 7 runs after a warm-up, 64 decoded tokens, greedy. llama.cpp build 11429 with the same GGUF files, measured in the same run. The machine carried background load, so compare within a run, not across runs.

Decode, Q8_0, tokens per second
ContextBeforeAlpacallama.cppAlpaca / llama.cpp
164565494301.28×
1284135133841.34×
5123304694011.17×
1,0242544984041.23×
2,0481774123541.16×
4,0961093783221.17×
Prefill, Q8_0, tokens per second
PromptBeforeAlpacallama.cppAlpaca / llama.cpp
1285,56317,35117,1631.01×
5123,99423,86425,5810.93×
2,0481,54117,04118,7290.91×
4,09684412,25614,8420.83×

Where the time went

The baseline prefill of a 2,048-token prompt took 1,270 ms on the GPU, and 87.5% of it was attention: a scalar kernel doing quadratic work and re-reading each key and value once per query head. The tiled kernel above is what fixed it.

Decode at 2,048 tokens of context spent 65% of each 5.5 ms token in attention, reading the cache at about 13 GB/s because a layer had only nine units of work. Split-K attention and GPU-resident greedy decoding are what moved those numbers.

Not everything worked, and the repository keeps the failures. Half-precision GEMM operands gained 1%. Pre-encoded decode steps gained nothing. A tensor-op attention prototype was slower than the shipped kernel, 1.84 ms against 1.61 ms per layer call. All three were measured and reverted. A memory audit also caught a dequantisation buffer, sized for the output table, that wasted 113 MiB per session in an early version.

Every operation has a CPU reference and a test against independently generated vectors with a stated tolerance. The CPU backend matches Hugging Face float32 logits to 5e-5 (F16) and 7e-5 (Q8_0), and all 191 tokenizer test strings produce identical token IDs.

03/Limits

  • Llama-family decoders only, with no RoPE scaling, sliding-window attention or mixture of experts.

  • F16, Q8_0 and Q4_0 weights. K-quants, Q5 and BF16 are rejected, and activations are not quantised.

  • One sequence per generation: no continuous batching, paged attention or speculative decoding. Prefill is 7 to 20% slower than llama.cpp at 512 to 4,096 tokens on quantised files.

  • Measured on one machine and one 135M-parameter model. Nothing has run on a physical iPhone or iPad yet.

SourceView on GitHub