plato

开源

腾讯高性能分布式图计算框架Plato

1.9K stars0 点赞1 次浏览

工具介绍

腾讯高性能分布式图计算框架Plato

Plato(柏拉图)

A framework for distributed graph computation and machine learning at wechat scale, for more details, see 柏拉图简介 | Plato Introduction.

Authors(In alphabetical order): Benli Li, Conghui He, Donghai Yu, Pin Gao, Shijie Sun, Wenqiang Wu, Wanjing Wei, Xing Huang, Xiaogang Tu, Yangzihao Wang, Yongan Li.

Contact: [email protected]

Special thanks to Xiaowei Zhu and many for their work Gemini[1]. Several basic utility functions in Plato is derived from Gemini, the design principle of some dual-mode based algorithms in Plato is also heavily influenced by Gemini's dualmode-engine. Thanks to Ke Yang and many for their work KnightKing[2] which served as foundation of plato's walk-engine.

Dependencies

To simplify installation, Plato currently downloads and builds most of its required dependencies by calling following commands. You should call it at least once before any build operations.

bash

# install compile dependencies.
sudo ./docker/install-dependencies.sh
# download and build staticlly linked libraries.
./3rdtools.sh distclean && ./3rdtools.sh install

Environment

Plato was developed and tested on x86_64 cluster and Centos 7.0. Theoretically, it can be ported to other Linux distribution easily.

Test && Build

bash

./build.sh

Run

Local

bash

./scripts/run_pagerank_local.sh

Production

Prerequisite:

A cluster which can submit MPI programs(Hydra is a feasible solution).
An accessible HDFS where Plato can find its input and put output on it.

A sample submit script was locate in here, modify it based on your cluster's environment and run.

bash

./scripts/run_pagerank.sh

Documents

Reference

[1] Xiaowei Zhu, Wenguang Chen, Weimin Zheng, Xiaosong Ma. Gemini: A computation-centric distributed graph processing system. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16)

[2] Ke Yang, Mingxing Zhang, Kang Chen, Xiaosong Ma, Yang Bai, Yong Jiang. KnightKing: A Fast Distributed Graph Random Walk Engine. In ACM SIGOPS 27th Symposium on Operating Systems Principles (SOSP ’19)