Recently, the developer Cactus Compute unveiled a streamlined AI model called Needle 2. This model boasts an incredibly compact size of just 14MB and is tailored for edge devices such as the Raspberry Pi 5. Notably, it can function without the necessity of AI acceleration expansion boards. Needle 2 has the capability to convert natural language commands into structured function calls in real-time at the local level. For instance, it can transform the command “dim the living room lights to 10%” into precise function parameters, all the while eliminating the need for a cloud connection to safeguard user privacy.

The model employs the Simple Attention Network architecture along with Cactus Quants quantization technology. This combination enables an extremely lightweight design without compromising on high performance. Needle 2 is compatible with a broad spectrum of platforms and operating systems. It supports architectures like ARM64, x86-64, RISC-V, and WebAssembly, as well as systems such as Apple, Windows, Linux, and Android.

Thanks to its self-contained design, the model can be directly embedded into binary files, free from runtime dependencies or network connections. This feature makes it effortless to integrate into various applications. When it comes to performance, Needle 2 showcases remarkable efficiency on resource-constrained devices. It achieves a decoding speed of up to 500 tokens per second on the Raspberry Pi 5 and between 300 - 700 tokens per second on low-end smartphones.