Zyphra's ZAYA1-8B: An Open MoE Reasoning Model With 760M Active Parameters That Runs on AMD
Zyphra released ZAYA1-8B under Apache 2.0 — a mixture-of-experts model with 8.4B total parameters but only 760M active per forward pass, trained entirely on AMD Instinct MI300X hardware, that matches DeepSeek-V3.2 and approaches frontier reasoning models on math benchmarks.
Zyphra released ZAYA1-8B on May 6, 2026: a mixture-of-experts language model that activates only 760M of its 8.4B total parameters per forward pass, was trained end-to-end on AMD hardware, and ships under Apache 2.0.
The active-parameter count is the headline. At inference time, ZAYA1-8B uses less compute than a 1B-parameter dense model while punching well above its apparent weight. On the APEX-shortlist benchmark it matches or exceeds DeepSeek-V3.2 and GPT-OSS-120B. On challenging math benchmarks it approaches DeepSeek-R1-0528, Gemini-2.5-Pro, and Claude 4.5 Sonnet. For a freely available model you can run on mid-range hardware, those are striking numbers.
Architecture. ZAYA1-8B introduces three non-standard components. Compressed Convolutional Attention (CCA) replaces standard multi-head attention with a convolutional compression step that reduces key/value sequence length before attending — cutting memory bandwidth for long contexts. An MLP-based expert router replaces the typical linear routing layer, giving richer routing decisions at the cost of a small added overhead. Learned residual scaling adjusts the contribution of each layer dynamically at training time, which Zyphra says stabilizes training at scale and improves final accuracy.
Test-time compute. The model pairs with Zyphra’s “Markovian RSA” inference method, which generates multiple reasoning traces in parallel and chunks context into fixed-length windows. The design delivers unbounded reasoning depth at constant memory cost — unlike chain-of-thought approaches where KV cache grows with the length of the reasoning trace.
Training stack. The entire run used a 1,024-GPU cluster of AMD Instinct MI300X accelerators on IBM Cloud, connected with AMD Pensando Pollara networking. Training ZAYA1-8B entirely on AMD hardware without falling back to NVIDIA GPUs at any stage is itself a statement — it demonstrates that AMD’s MI300X ecosystem has matured enough for serious frontier model work.
Availability. ZAYA1-8B is available now as a free serverless API endpoint on Zyphra Cloud and as a full model download on Hugging Face. The Apache 2.0 license means commercial use is unrestricted.
For developers building reasoning-heavy applications, the math is favorable: a model that delivers near-frontier reasoning at 760M active parameters is a meaningful step toward practical deployment on inference-constrained hardware. Whether ZAYA1-8B’s gains hold across broader benchmarks remains to be tested, but the architecture choices — particularly CCA and Markovian RSA — are thoughtful enough to earn close attention from the open-source ML community.
Zyphra is a relatively young lab. ZAYA1-8B is their most ambitious public release to date, and the decision to train on AMD at scale reflects both commercial positioning (differentiation from the CUDA monoculture) and genuine capability in hardware-agnostic training pipelines.