Show HN: Gemma 4 26B at 100 tok/s on a Mac, using the Neural Engine and 2GB RAM

  • Posted 1 hour ago by daemon1123
  • 1 points
https://github.com/zw5/ane-fieldfare
After checking out Turbo Fieldfare (a very interesting project that streamed experts from SSD on Apple Silicon Macs to run Gemma 4 26B with very little RAM usage), I tried to make this project much faster and succeeded!

This project gets up to 100+ tokens/s inference on Gemma 4 26B (Turbo Fieldfare runs at 20-30 on similar hardware) and doesn't have insanely slow prefill times due to the ANE helping (I had to reverse engineer Apple's APIs to get this working).

I also implemented speculative decoding, and while Turbo steered away from using mmap, this repo utilizes it so the OS file cache serves as the expert cache.

0 comments