RISC-V Steps Up: Running Large AI Models Without GPUs

At the 2026 RISC-V Ecosystem Conference in Shanghai, Alibaba's DAMO Academy demonstrated something no chip using the open-source RISC-V instruction set had ever done before: running a 27-billion-parameter language model entirely on a CPU, with no GPU, no emulation layer, and no translation overhead. The chip was Alibaba's XuanTie C950, a 64-core RISC-V server processor built on TSMC's 5nm process.

The model was Qwen 3.8 27B — released under an Apache 2.0 license on August 14, 2026. On the C950, it achieved 30 tokens per second decode speed with a time-to-first-token of 1.9 seconds. For context, running a 27-billion-parameter model conventionally requires 16 to 24 GB of dedicated GPU memory. The C950 handled the same inference task using its CPU cores and integrated AI acceleration engines alone.

How it works: integrated AI acceleration

What distinguishes the C950 from a conventional server CPU is its integrated AI acceleration. Alibaba embedded two purpose-built engines directly into the chip: a Vector Acceleration Engine for parallel data operations and a Matrix Acceleration Engine for the tensor computations that dominate transformer-based inference.

These are not bolted-on co-processors. They share unified addressing with the CPU cores, creating tight coupling between general-purpose compute and AI-specific workloads. At the heart of the AI subsystem sits the XuanTie Tensor Processing Engine (TPE), delivering 8 TOPS per unit and supporting data formats critical for modern AI: FP16, FP8, INT4, and microscaling formats like MXFP8, MXFP4, and RVFP4.

The chip also implements the RISC-V Attached Matrix Extension (AME), a proprietary ISA extension that lets the matrix engine execute tensor operations directly through the CPU's instruction pipeline — blurring the line between general-purpose compute and AI accelerator in a way that is architecturally distinctive.

Why this demonstration matters

The immediate implications are practical. Edge inference — in factories, retail environments, autonomous vehicles, and robotics platforms — often lacks the power, cooling, or physical space for discrete GPU hardware. A C950-based system can run LLM inference alongside standard cloud workloads like MySQL, Redis, and Nginx on the same silicon, eliminating the need for separate GPU-accelerated inference servers.

Cost and energy efficiency improve substantially. GPU inference for a 27B model typically consumes hundreds of watts per accelerator. The C950's CPU-centric approach, with tightly integrated acceleration engines, delivers inference at a fraction of the power envelope. For cloud providers running mixed workloads, the ability to serve both general-purpose computing and AI inference on the same processor type simplifies infrastructure and reduces hardware diversity.

For AI sovereignty, the C950 is strategically significant. RISC-V offers up to 4× lower power consumption and enables custom AI acceleration without licensing constraints. Alibaba can customize the chip for specific AI workloads, avoid dependency on Western intellectual property that could be restricted by export controls, and build a vertically integrated hardware-software stack that no foreign government can throttle.

The broader RISC-V story

The C950 arrives at a pivotal moment for RISC-V. In 2026, the open-source instruction set architecture crossed the 25% global market share threshold across application processors, microcontrollers, and AI accelerators. For the first time in 25 years, the x86 and ARM duopoly has become a three-way contest.

The SHD Group projects 36 billion RISC-V device shipments by 2031, with a market value exceeding $300 billion. Recent RISC-V designs have closed the performance gap with ARM — the Tenstorrent Ascalon-X core achieves roughly 22 SPECint2006 points per GHz, placing it in direct parity with AMD's Zen 5 and ARM's Neoverse V3. Studies indicate that RISC-V designs deliver 30 to 40% better power-performance-area metrics than comparable ARM configurations for cloud-native workloads.

Meanwhile, SiFive — the company founded by RISC-V's creators — introduced the BigSky SF-2U870, the world's first enterprise-grade, rackable 2U RISC-V server, demonstrated at the same Hot Chips 2026 event. The platform is being used to port CUDA to RISC-V hardware in collaboration with NVIDIA, integrating NVLink Fusion into future SiFive platforms.

Tradeoffs

The 30 tokens per second throughput on the C950 is significantly lower than what a high-end GPU like an NVIDIA H100 can achieve on the same model. The C950 runs a single inference thread per socket, making it better suited for edge deployment and private inference than for high-concurrency public API serving. For latency-sensitive agentic workloads and interactive applications, where GPU hardware is unavailable or impractical, the C950 offers a viable and increasingly competitive alternative — but it is not replacing GPUs for large-scale public inference.

The real story is optionality. RISC-V's open standard with no royalties and no gatekeeper gives designers the freedom to extend the base ISA with custom extensions for AI, DSP, cryptography, or domain-specific workloads. As the performance gap with ARM and x86 continues to close, that flexibility — combined with zero licensing cost and no supply-chain dependency — is becoming increasingly attractive for custom AI silicon.