What happened
The wici-ai/inferssd GitHub repository describes InferSSD as a way to run large mixture-of-experts (MoE) models from standard GGUF files by streaming expert weights from SSD. It is built on llama.cpp and offers a terminal chat interface, a browser UI, and an OpenAI-compatible API. The repository lists macOS with Metal and Linux with CUDA as supported platforms; Windows 11 x64 packages are also described, while Windows support is labeled in progress. The project says its numbered releases include checked prebuilt packages for Apple Silicon macOS 15 or later and Windows 11 x64. Linux builds from source for the machine’s CPU and GPU, a process the repository estimates at two to 12 minutes.
The project’s tested GGUF model list includes DeepSeek-V4.1-Flash-1017, with 763 billion total and 8–16 billion active parameters; Muse-Spark-Open, with 816 billion total and 42 billion active; Qwen3.6-35B-A3B; Qwen3.8 Flash Next; and Gemma4 E4B. The repository identifies the GGUF files as community conversions of the named model releases. It notes that Gemma uses the ordinary native loader, without SSD expert offload. For DeepSeek, it says the Q8_0 conversion keeps official bit widths: routed experts remain native 4-bit MXFP4, while FP8 tensors become 8-bit Q8_0.
The benchmark table specifies a 4K prompt tier and 256 output tokens, with decode measured over tokens 129–256. On a MacBook Pro with an M5 Max, 48 GB unified memory, and a 2 TB internal SSD, InferSSD’s listed DeepSeek decode rate is 2.874 tokens per second, compared with 0.873 for llama.cpp using the same hardware. On an RTX 5060 Ti system with 8 GB GPU memory, 16 GB VRAM as stated in the table, and a 2 TB PCIe 4 drive, the listed rate is 2.550 tokens per second versus 0.242 for llama.cpp. These are repository-reported measurements, not an independent evaluation.
The documented installation command downloads a release package on supported platforms; Linux users build from source. A first run downloads model files to a local models directory and measures the machine. The repository estimates about 470 GiB for DeepSeek or Muse and 21 GiB for Qwen3.6. Users can choose a model directory or point the launcher at existing files. The project says it checks whether weights, KV cache, and working memory fit its memory budget and rejects a model that does not fit. Its OpenAI-compatible API listens locally by default; the repository explicitly warns that it has no authentication, so anyone who can reach a port exposed with --host 0.0.0.0 can use it.
What the archive adds
New Runtime’s earlier coverage includes Slotstream’s SSD-streaming approach for 125B MoE models on Macs (2026-10-27) and Betelgeuse’s NVMe-paged inference for a 125B model (2026-10-31). InferSSD is a new project in a recurring technical thread: using storage to extend the hardware envelope for local MoE inference, with its own benchmark set and model targets.
Trend signal
This does not yet fit a pattern New Runtime tracks. The project is a concrete systems implementation for local inference, but the current pattern set does not describe SSD offload or local hardware thresholds closely enough to claim that it materially supports one of those broader theses.
Why now
The repository’s current model list spans several large MoE releases, while its benchmark table attempts to show how the same runtime behaves across Apple Silicon, consumer NVIDIA GPUs, and a DGX Spark. That makes the project timely for builders evaluating whether storage-backed inference can make particular models usable on hardware they already own, rather than assuming model size alone decides feasibility.
Why it matters
InferSSD makes a useful distinction between a model’s total parameter count and the smaller active portion of an MoE model, while also showing the cost of moving expert data from storage during inference. That can broaden the set of experiments possible on a workstation without claiming that SSD streaming makes a huge model behave like a small one. The repository’s reported decode rates remain low in absolute terms on some configurations, and the required model files can consume hundreds of gigabytes. For teams, the value is therefore not a blanket replacement for hosted inference; it is another point on the trade-off curve among memory capacity, SSD capacity, throughput, and data locality. The exposed API also creates an operational responsibility: the lack of authentication matters if the service is reachable beyond the local machine.
Limits and uncertainty
The performance figures are published by the project itself; the repository excerpt does not establish independent reproduction or provide evidence about variance across repeated runs. Its benchmark setup is specific to listed hardware, a 4K prompt tier, and a 256-token output, so results may not transfer to other drives, model conversions, prompts, or workloads. The repository identifies tested community-converted GGUF files, but the performance and behavior of other conversions are not established here. Practical usability also depends on model download size, memory available after the system’s other demands, and acceptable response latency. The API’s lack of authentication is an explicit security limitation, not a deployment detail to overlook.
Skill to build now
If you are assessing local MoE inference, make a small test plan before downloading a model: record the exact GGUF conversion, hardware, available memory, SSD capacity, prompt length, output length, and decode speed. Compare results only across runs with matching conditions, and treat the repository’s table as a starting hypothesis rather than a capacity guarantee. Keep the API bound to localhost unless you add an access-control layer appropriate to your environment.
Reader takeaway
InferSSD offers a practical experiment in trading SSD bandwidth for memory capacity when running large MoE models locally. Its published results suggest that the approach can work on several classes of hardware, but “fits” does not mean fast: check the exact model size and measured decode rate against your workload, and do not expose its unauthenticated API to an untrusted network.