StepMesh: A High-Performance, Low-Latency Communication Library for Attention-FFN Disaggregation StepMesh is a communication library design…
StepMesh: A High-Performance, Low-Latency Communication Library for Attention-FFN Disaggregation StepMesh is a communication library design…
StepMesh is a communication library designed to provide high-performance and low-latency communication for Attention-FFN decoupling architectures.
The codebase is developed upon BytePS.
The following diagram illustrates the architecture of the StepMesh communication framework with the follow highlights.
Unified and Flexible ZeroCopy PushPull Interface: StepMesh provides a single, efficient interface (ZBatchPushPull) for tensor transfer. ZBatchPushPull interface allows users to tailor communication strategies to specific application needs.
Communication Across Multiple Accelerators: Designed to work with different computing Accelerators, StepMesh leverages RDMA to achieve high-speed, low-latency communication and is architected to easily incorporate support for new computing chips.
Low Latency: StepMesh is engineered to provide the ultra-low latency communication essential for Attention-FFN decoupling architectures. Its performance is on par with the limits imposed by hardware components.
AFTensorWorker API: This API is responsible for managing the communication operations performed by attention nodes. It provides methods such as Wait and PushPull for synchronization and data transfer.
AFTensorServer API: This API handles the communication operations performed by FFN nodes. It includes methods like GetBatch and Respond for retrieving and responding to requests from workers.
PushPull method to send and receive data to/from servers. The Wait method is used for synchronization, ensuring that data transfers are completed before proceeding with further computations.GetBatch method to retrieve data from workers and the Respond method to send data back to workers. These methods facilitate efficient data exchange between servers and workers.Download code and install dependencies
git clone https://github.com/stepfun-ai/StepMesh
cd StepMesh
bash tools/install_deps.sh # only onceBuild StepMesh
# Please check that your CUDA_HOME is correct and torch is installed.
# Build AF library
make af
# Build and install Fserver (AF's Python SDK)
pip3 install -v -e . --no-build-isolationBuild without cuda
# Build AF library
USE_CUDA=0 make af
# Build and install Fserver (AF's Python SDK)
USE_CUDA=0 pip3 install -v -e . --no-build-isolation
In StepMesh, there are three roles: worker, server and scheduler. Each role is an independent process.
The scheduler is responsible for setting up the connections between workers and servers at initialization. There should be only 1 scheduler process.
A worker process only communicates with server processes, and vice versa. There won't be any traffic between worker-to-worker, and server-to-server.
Workers can push or pull tensors to servers, and all operations are async. For pushing tensors, the tensors are broadcasted to servers
After build, you will have testing applications under tests/ dir.
Below we elaborate how you can run with them.
To debug, set PS_VERBOSE=1 to see important logs during connection setup, and PS_VERBOSE=2 to see each message log.
# ROLE: jointly run scheduler, worker and server; RNIC: your first rdma nic;
ROLE=joint RNIC=brainpf_bond0 bash tests/fserver/run_single_gpu.sh# ROLE: jointly run scheduler, worker and server; RNIC: your first rdma nic;
ROLE=joint RNIC=brainpf_bond0 bash tests/fserver/run_multi_gpu.sh# server: RNIC: your first rdma nic;
ROLE=server RNIC=brainpf_bond0 bash tests/fserver/run_multi_gpu.sh
# the first line will print scheduler ip used for worker# worker: RNIC: your first rdma nic;
ROLE=worker RNIC=brainpf_bond0 bash tests/fserver/run_multi_gpu.sh ${scheduler ip}For more test cases and examples, please refer to tests.
For more documents, please refer to docs.
For more details, please refer to the Step-3 system technical report and our Introduction(Chinese Version).
No open issues yet, or sync has not completed.