Spiking Brain-inspired Large Models, integrating hybrid efficient attention, MoE modules and spike encoding into its architecture
Spiking Brain-inspired Large Models, integrating hybrid efficient attention, MoE modules and spike encoding into its architecture
Technical Report: Chinese | English
Arxiv: arXiv:2509.05276
Models: Available Models
Demo: OpenBayes贝式计算
Inspired by brain mechanisms, SpikingBrain integrates hybrid efficient attention, MoE modules, and spike encoding into its architecture, supported by a universal conversion pipeline compatible with the open-source model ecosystem. This enables continual pre-training with less than 2% of the data while achieving performance comparable to mainstream open-source models. We further adapt frameworks, operators, parallel strategies, and communication primitives for non-NVIDIA (MetaX) clusters, ensuring stable large-scale training and inference. SpikingBrain achieves over 100× speedup in TTFT for 4M-token sequences, while spiking delivers over 69% sparsity at the micro level. Combined with macro-level MoE sparsity, these advances provide valuable guidance for the design of next-generation neuromorphic chips.
This repository provides the full implementation and weights of SpikingBrain-7B, including the HuggingFace version, vLLM inference version, and quantized version, enabling flexible deployment and research across different scenarios.
SpikingBrain-7B/
├── hf_7B_model/ # HuggingFace version
├── hf_7B_VLM/ # VLM Model HuggingFace version
├── run_model/ # Model run examples
├── vllm_hymeta/ # vLLM plugins and inference support
├── W8ASpike/ # Quantized inference version
├── setup.py
├── requirements.txt
└── README.md vllm-hymeta is the plugin adaptation of HyMeta (Hybrid Models built on MetaX GPUs) for the vLLM inference framework, providing efficient inference support on NVIDIA GPUs.
By leveraging the plugins mechanism in vLLM, hardware backends can be integrated in a modular fashion, bringing the following benefits:
Decoupled codebase: Backend-specific code remains independent, keeping the vLLM core cleaner.
Reduced maintenance cost: vLLM developers can focus on general functionality without being affected by backend-specific implementations.
Faster integration: New backends can be integrated quickly and evolve independently with less engineering effort.
git clone https://github.com/BICLab/SpikingBrain-7B.git
cd SpikingBrain-7B
docker build -t spiking-brain:7b-v1.0 .docker run -itd \
--entrypoint /bin/bash \
--network host \
--name \
--shm-size 160g \
--gpus all \
--privileged \
-v /host_path:/container_path \
spiking-brain:7b-v1.0Recommended environment for installing vllm-hymeta on NVIDIA GPUs:
torch==2.7.1
transformers==4.55.2
triton==3.3.1
flash_attn==2.7.3
flash-linear-attention==0.1
vllm==0.10.0
scipy
pyyaml
decorator
setuptools
setuptools-scmYou can serve a model with vLLM in the simplest way using the following command:
vllm serve \
--served-model-name \
--gpu-memory-utilization \
--block-size \
--dtype bfloat16 \
--port You may also set --tensor-parallel-size and --pipeline-parallel-size when launching if you want to run with multiple GPUs.
W8ASpike is the quantized inference version of SpikingBrain-7B, aiming to reduce inference cost under low-precision settings and explore the potential of Spiking Neural Networks (SNNs).
The current implementation adopts pseudo-spiking, where activations are approximated as spike-like signals at the tensor level, rather than true asynchronous event-driven spiking on neuromorphic hardware.
Pseudo-spiking: Efficient approximation at the tensor level, suitable for prototyping and research.
True-spiking: Requires asynchronous hardware and event-driven operator support, which is beyond the scope of this repository.
The activation spike encoding process here is inspired by the pseudo-spiking interfaces from BICLab/Int2Spike. For additional PyTorch-based spiking interfaces, please refer to the Int2Spike library.
The model weights are hosted on ModelScope. Please select the appropriate version based on your needs:
Example scripts are provided in run_model/ for running the model with the released checkpoints.
Hugging Face
Load with AutoModelForCausalLM and use as a standard CausalLM (forward or generation); see run_model/run_model_hf.py.
For the SFT model, a chat template is used; see run_model/run_model_hf_chat_template.py.
vLLM
Perform inference using the provided vLLM Hymeta plugin; see run_model/run_model_vllm.py and the vLLM Hymeta section.
Please make sure to remove the auto_map field from config.json. Specifically, delete the following block if it is present:
"auto_map": {
"AutoConfig": "configuration_gla_swa.GLAswaConfig",
"AutoModelForCausalLM": "modeling_gla_swa.GLAswaForCausalLM"
}For tested environment, please refer to requirements.txt.
Table 1: Performance evaluation of the SpikingBrain-7B pre-trained model. All models are tested with the HuggingFace framework and evaluated using a perplexity-based method. Except for Qwen2.5, the other baselines are trained on limited Chinese data, resulting in clear disadvantages on CMMLU and C-Eval.
Table 2: Performance evaluation of the SpikingBrain-76B pre-trained model. All models are tested with the vLLM framework and evaluated using a perplexity-based method. Except for Qwen2.5, the other baselines are trained on limited Chinese data, resulting in clear disadvantages on CMMLU and C-Eval.
If you find our work useful, please consider citing SpikingBrain:
@article{pan2025spikingbrain,
title={SpikingBrain: Spiking Brain-inspired Large Models},
author={Pan, Yuqi and Feng, Yupeng and Zhuang, Jinghao and Ding, Siyu and Xu, Han and Liu, Zehao and Sun, Bohan and Chou, Yuhong and Qiu, Xuerui and Deng, Anlin and others},
journal={arXiv preprint arXiv:2509.05276},
year={2025}
}
No open issues yet, or sync has not completed.