百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
P

pipedream

> 编程语言
开源

PipeDream: DNN 训练的管道并行化 此仓库包含以下论文的源代码实现: - "PipeDr…

395 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

PipeDream: DNN 训练的管道并行化 此仓库包含以下论文的源代码实现: - "PipeDr…

PipeDream: Pipeline Parallelism for DNN Training

This repository contains the source code implementation of the following papers:

  • "PipeDream: Generalized Pipeline Parallelism for DNN Training", which appeared at SOSP 2019 (pipedream branch).
  • "Memory-Efficient Pipeline-Parallel DNN Training", which appeared at ICML 2021 (pipedream_2bw branch).

This work was one as part of Microsoft Research's Project Fiddle. This source code is available under the MIT License.

Directory Structure

graph

This contains a Python implementation of a graph, used by the PipeDream profiler and optimizer. Profiling scripts in profiler generate graph profiles, that can then be ingested by the optimizer located in optimizer to generate a partitioned model, that can then be fed to the PipeDream runtime.

profiler

Instrumented PyTorch applications which return profiles that can be ingested by the optimizer.

optimizer

A Python implementation of PipeDream's optimizer.

runtime

PipeDream's runtime, which implements model parallelism, as well as input pipelining in PyTorch. This can be fused with data parallelism to give hybrid model and data parallelism, and input pipelining.

Setup

Software Dependencies

To run PipeDream, you will need a NVIDIA GPU with CUDA 10.0, GPU driver version 418.56, nvidia-docker2, and Python 3. On a Linux server with NVIDIA GPU(s) and Ubuntu 16.04, these dependencies can be installed using,

bash
bash setup.sh

All dependencies are in the nvcr.io/nvidia/pytorch:19.05-py3 container, which can be downloaded using,

bash
nvidia-docker pull nvcr.io/nvidia/pytorch:19.05-py3

To run the PipeDream profiler, you will need to build a new Docker image, which can be done using the Dockerfile in this directory. Note that the Dockerfile has a dependency on the pre_hook.patch and requirements.txt files in this directory. This container can be built using,

bash
docker build --tag  .

The PyTorch Docker Container can then be run using,

bash
nvidia-docker run -it -v /mnt:/mnt --ipc=host --net=host  /bin/bash

Data

Image Classification

All image classification experiments are run using the ImageNet ILSVC 2012 dataset. This can be downloaded using the following command (within the docker container above),

bash
cd scripts; python download_imagenet.py --data_dir 

Note that the ImageNet dataset is about 145GB, so this download script can take some time.

Translation

All translation experiments are run using the WMT En-De dataset, also used for the MLPerf translation (RNN) task. This can be downloaded using the instructions in the MLPerf repository.

End-to-end Workflow

To run a demo, run the following commands (the optimizer and runtime have been verified to work unchanged in nvcr.io/nvidia/pytorch:19.05-py3). More detailed instructions for each of the individual components are in the corresponding directory READMEs, and more detailed instructions on how to run the main experiments in the SOSP paper are in EXPERIMENTS.md.

[from pipedream/profiler/image_classification; you will need to have the changes to PyTorch listed above] Note that the profiling step must be run with only a single GPU (hence the CUDA_VISIBLE_DEVICES=0 before the command).

bash
CUDA_VISIBLE_DEVICES=0 python main.py -a vgg16 -b 64 --data_dir 

[from pipedream/optimizer]

bash
python optimizer_graph_hierarchical.py -f ../profiler/image_classification/profiles/vgg16/graph.txt -n 4 --activation_compression_ratio 1 -o vgg16_partitioned

[from pipedream/optimizer]

bash
python convert_graph_to_model.py -f vgg16_partitioned/gpus=4.txt -n VGG16Partitioned -a vgg16 -o ../runtime/image_classification/models/vgg16/gpus=4 --stage_to_num_ranks 0:3,1:1

[from pipedream/runtime/image_classification; run on 4 GPUs (including a single server with 4 GPUs)]

…

master IP address here is the IP address of the rank 0 process. On a server with 4 GPUs, localhost can be specified.

When running DP setups, please use the nccl backend for optimal performance. When running hybrid setups, please use the gloo backend.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

License

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT license.

Issues· 33 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言