GraphScope

A One-Stop Large-Scale Graph Computing System from Alibaba

See our ongoing [GraphScope Flex](https://github.com/alibaba/GraphScope/tree/main/flex): a LEGO-inspired, modular, and user-friendly GraphScope evolution. GraphScope is a unified distributed graph computing platform that provides a one-stop environment for performing diverse graph operations on a cluster of computers through a user-friendly Python interface. GraphScope makes multi-staged processing of large-scale graph data on compute clusters simply by combining several important pieces of Alibaba technology: including [GRAPE](https://github.com/alibaba/libgrape-lite), [MaxGraph](interactive_engine/), and [Graph-Learn](https://github.com/alibaba/graph-learn) (GL) for analytics, interactive, and graph neural networks (GNN) computation, respectively, and the [Vineyard](https://github.com/v6d-io/v6d) store that offers efficient in-memory data transfers. Visit our website at [graphscope.io](https://graphscope.io) to learn more. ## Latest News - [21/04/2025] GraphScope achieved record-breaking results on the [LDBC Social Network Benchmark Interactive workload](https://ldbcouncil.org/benchmarks/snb-interactive/) using declarative query language CYPHER, with a 2.0× higher throughput on SF300 than the previous record holder! - [31/07/2024] We’ve launched a webpage visualizing GraphScope’s journey in graph computing. [Check it out!](https://graphscope.io/journey) - [30/05/2024] GraphScope Flex set new record-breaking [SNB Interactive audit results](https://ldbcouncil.org/benchmarks/snb-interactive/), as announced by LDBC on [X (Twitter)](https://twitter.com/LDBCouncil/status/1795886732950294630)! - [25/03/2024] We donated the graph file format [GraphAr](https://graphar.apache.org/) to [Apache Software Foundation](https://www.apache.org/) as an Incubating Project. - [05/02/2024] GraphScope Flex [paper](https://arxiv.org/abs/2312.12107) was accepted by [SIGMOD 2024](https://2024.sigmod.org/) Industry Track. See you in ! - [19/12/2023] A paper introducing GraphScope Flex released on [arXiv.org](https://arxiv.org/abs/2312.12107). - [20/07/2023] GraphScope achieved record-breaking results on the [LDBC Social Network Benchmark Interactive workload](https://ldbcouncil.org/benchmarks/snb-interactive/), with a 2.45× higher throughput on SF300 than the previous record holder! - [04/07/2023] GraphScope Flex tech preview released with [v0.23.0](https://github.com/alibaba/GraphScope/releases/tag/v0.23.0). ## Table of Contents - [Getting Started](#getting-started) - [Installation for Standalone Mode](#installation-for-standalone-mode) - [Demo: Node Classification on Citation Network](#demo-node-classification-on-citation-network) - [Loading a graph](#loading-a-graph) - [Interactive query](#interactive-query) - [Graph analytics](#graph-analytics) - [Graph neural networks (GNNs)](#graph-neural-networks-gnns) - [Graph Processing on Kubernetes](#processing-large-graph-on-kubernetes-cluster) - [Creating a session](#creating-a-session) - [Loading graphs and graph computation](#loading-a-graph-and-processing-computation-tasks) - [Closing the session](#closing-the-session) - [Development](#development) - [Building from source](#building-on-local) - [Building Docker images](#building-docker-images) - [Building the client library](#building-client-library) - [Testing](#testing) - [Documentation](#documentation) - [License](#license) - [Publications](#publications) - [Joining our Community!](#contributing) ## Getting Started We provide a [Playground](https://try.graphscope.io) with a managed JupyterLab. [Try GraphScope](https://try.graphscope.io) straight away in your browser! GraphScope supports running in standalone mode or on clusters managed by [Kubernetes](https://kubernetes.io/) within containers. For quickly getting started, let's begin with the standalone mode. ### Installation for Standalone Mode GraphScope pre-compiled package is distributed as a python package and can be easily installed with `pip`. ```bash pip3 install graphscope ``` Note that `graphscope` requires `Python` >= `3.8` and `pip` >= `19.3`. The package is built for and tested on the most popular Linux (Ubuntu 20.04+ / CentOS 7+) and macOS 12+ (Intel/Apple silicon) distributions. For Windows users, you may want to install Ubuntu on WSL2 to use this package. Next, we will walk you through a concrete example to illustrate how GraphScope can be used by data scientists to effectively analyze large graphs. ## Demo: Node Classification on Citation Network [`ogbn-mag`](https://ogb.stanford.edu/docs/nodeprop/#ogbn-mag) is a heterogeneous network composed of a subset of the Microsoft Academic Graph. It contains 4 types of entities(i.e., papers, authors, institutions, and fields of study), as well as four types of directed relations connecting two entities. Given the heterogeneous `ogbn-mag` data, the task is to predict the class of each paper. Node classification can identify papers in multiple venues, which represent different groups of scientific work on different topics. We apply both the attribute and structural information to classify papers. In the graph, each paper node contains a 128-dimensional word2vec vector representing its content, which is obtained by averaging the embeddings of words in its title and abstract. The embeddings of individual words are pre-trained. The structural information is computed on-the-fly. ### Loading a graph GraphScope models graph data as property graph, in which the edges/vertices are labeled and have many properties. Taking `ogbn-mag` as example, the figure below shows the model of the property graph. This graph has four kinds of vertices, labeled as `paper`, `author`, `institution` and `field_of_study`. There are four kinds of edges connecting them, each kind of edges has a label and specifies the vertex labels for its two ends. For example, `cites` edges connect two vertices labeled `paper`. Another example is `writes`, it requires the source vertex is labeled `author` and the destination is a `paper` vertex. All the vertices and edges may have properties. e.g., `paper` vertices have properties like features, publish year, subject label, etc. To load this graph to GraphScope with our retrieval module, please use these code: ```python import graphscope from graphscope.dataset import load_ogbn_mag g = load_ogbn_mag() ``` We provide a set of functions to load graph datasets from [ogb](https://ogb.stanford.edu/docs/dataset_overview/) and [snap](https://snap.stanford.edu/data/index.html) for convenience. Please find all the available graphs [here](https://github.com/alibaba/GraphScope/tree/docs/python/graphscope/dataset). If you want to use your own graph data, please refer [this doc](https://graphscope.io/docs/loading_graph.html) to load vertices and edges by labels. ### Interactive query Interactive queries allow users to directly explore, examine, and present graph data in an *exploratory* manner in order to locate specific or in-depth information in time. GraphScope adopts a high-level language called [Gremlin](http://tinkerpop.apache.org/) for graph traversal, and provides [efficient execution](interactive_engine/benchmark/) at scale. In this example, we use graph traversal to count the number of papers two given authors have co-authored. To simplify the query, we assume the authors can be uniquely identified by ID `2` and `4307`, respectively. ```python # get the endpoint for submitting Gremlin queries on graph g. interactive = graphscope.gremlin(g) # count the number of papers two authors (with id 2 and 4307) have co-authored papers = interactive.execute("g.V().has('author', 'id', 2).out('writes').where(__.in('writes').has('id', 4307)).count()").one() ``` ### Graph analytics Graph analytics is widely used in real world. Many algorithms, like community detection, paths and connectivity, centrality are proven to be very useful in various businesses. GraphScope ships with a set of [built-in algorithms](https://graphscope.io/docs/analytics_engine.html#built-in-algorithms), enables users easily analysis their graph data. Continuing our example, below we first derive a subgraph by extracting publications in specific time out of the entire graph (using Gremlin!), and then run k-core decomposition and triangle counting to generate the structural features of each paper node. Please note that many algorithms may only work on *homogeneous* graphs, and therefore, to evaluate these algorithms over a property graph, we need to project it into a simple graph at first. ```python # extract a subgraph of publication within a time range sub_graph = interactive.subgraph("g.V().has('year', gte(2014).and(lte(2020))).outE('cites')") # project the projected graph to simple graph. simple_g = sub_graph.project(vertices={"paper": []}, edges={"cites": []}) ret1 = graphscope.k_core(simple_g, k=5) ret2 = graphscope.triangles(simple_g) # add the results as new columns to the citation graph sub_graph = sub_graph.add_column(ret1, {"kcore": "r"}) sub_graph = sub_graph.add_column(ret2, {"tc": "r"}) ``` In addition, users can write their own algorithms in GraphScope. Currently, GraphScope supports users to write their own algorithms in Pregel model and PIE model. ### Graph neural networks (GNNs) Graph neural networks (GNNs) combines superiority of both graph analytics and machine learning. GNN algorithms can compress both structural and attribute information in a graph into low-dimensional embedding vectors on each node. These embeddings can be further fed into downstream machine learning tasks. In our example, we train a GCN model to classify the nodes (papers) into 349 categories, each of which represents a venue (e.g. pre-print and conference). To achieve this, first we launch a learning engine and build a graph with features following the last step. ```python # define the features for learning paper_features = [f"feat_{i}" for i in range(128)] paper_features.extend(["kcore", "tc"]) # launch a learning engine. lg = graphscope.graphlearn(sub_graph, nodes=[("paper", paper_features)], edges=[("paper", "cites", "paper")], gen_labels=[ ("train", "paper", 100, (0, 75)), ("val", "paper", 100, (75, 85)), ("test", "paper", 100, (85, 100)) ]) ``` Then we define the training process, and run it. ``` … ``` A Python script with the entire process is available [here](https://colab.research.google.com/github/alibaba/GraphScope/blob/main/tutorials/1_node_classification_on_citation.ipynb), you may try it out by yourself. ## Processing Large Graph on Kubernetes Cluster GraphScope is designed for processing large graphs, which are usually hard to fit in the memory of a single machine. With [Vineyard](https://github.com/v6d-io/v6d) as the distributed in-memory data manager, GraphScope supports running on a cluster managed by Kubernetes(k8s). To continue this tutorial, please ensure that you have a k8s-managed cluster and know the credentials for the cluster. (e.g., address of k8s API server, usually stored a `~/.kube/config` file.) Alternatively, you can set up a local k8s cluster for testing with [Kind](https://kind.sigs.k8s.io/). You can install and deploy Kind referring to [Quick Start](https://kind.sigs.k8s.io/docs/user/quick-start/); If you did not install the `graphscope` package in the above step, you can install a subset of the whole package with client functions only. ```bash pip3 install graphscope-client ``` Next, let's revisit the example by running on a cluster instead. The figure shows the flow of execution in the cluster mode. When users run code in the python client, it will: - *Step 1*. Create a session or workspace in GraphScope. - *Step 2 - Step 5*. Load a graph, query