Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
C

cluster-toolkit

> 编程语言
Open source

Cluster Toolkit is an open-source software offered by Google Cloud which makes it easy for customers to deploy AI/ML and HPC environments on Google Cloud.

353 stars0 likes1 views
WebsiteGitHub

About

Cluster Toolkit is an open-source software offered by Google Cloud which makes it easy for customers to deploy AI/ML and HPC environments on Google Cloud.

Google Cluster Toolkit (formerly HPC Toolkit)

Description

Cluster Toolkit is an open-source software provided by Google Cloud that makes it easy to deploy AI/ML and high performance computing (HPC) environments following Google Cloud best practices.

Cluster Toolkit is highly customizable and extensible, addressing the deployment needs of a broad range of workloads, such as compute, networking, and storage, in a repeatable manner.

Detailed documentation and main components

Cluster Toolkit comes with a suite of tutorials, examples, and full documentation for modules designed for AI/ML and HPC use cases.

The main components of Cluster Toolkit include:

  • Cluster blueprints: YAML files that define the cluster's infrastructure and configuration.
  • Modules: Reusable building blocks (Terraform or Packer) used to compose a blueprint.
  • gcluster engine: The command-line tool that processes blueprints to create a deployment folder.
  • Deployment folder: A self-contained folder containing the Terraform or Packer code needed to provision the environment.

For more information, see Google Cluster Toolkit overview.

AI Hypercomputer

Cluster Toolkit is an integral part of Google Cloud AI Hypercomputer. For more information about GKE and Slurm deployments in AI Hypercomputer, see Create an AI-optimized GKE cluster with default configuration and Create a self-managed Slurm cluster for AI workloads.

Quickstart

To get started with Cluster Toolkit deployments, you can follow one of the following quickstart guides:

  • Deploy an HPC cluster with Slurm.
  • Create a self-managed Slurm cluster with an A4 VM.
  • Create a Cloud RDMA-enabled HPC Slurm cluster with H4D instances.

Install Cluster Toolkit

To create a cluster using Cluster Toolkit, you can use either Cloud Shell, or a workstation that is running Linux or macOS.

Cloud Shell is an interactive development and operations environment that is accessible from your web browser. If you use Cloud Shell, the following dependencies are already pre-installed and you don't need to manually install dependencies.

If you want to work from a Linux or macOS client or workstation, you must follow the steps in Install dependencies before you continue.

You can use two different methods to install Cluster Toolkit:

  • Using the pre-built bundle (recommended)
  • Building from source

Using the pre-built bundle (recommended)

For the easiest setup, download the appropriate bundle for your operating system and architecture (e.g., gcluster_bundle_linux_amd64.zip, gcluster_bundle_linux_arm64.zip, gcluster_bundle_mac_amd64.zip, or gcluster_bundle_mac_arm64.zip) from the Releases page. These bundles include the pre-compiled gcluster binary, the examples folder, and the community/examples folder.

Bundle compatibility matrix

The pre-built bundles are compiled for Linux and macOS execution environments and support the deployment of the following cluster operating systems.

Execution platform (where the binary runs)
Platform Support Status Notes
Linux (amd64 / arm64) ✅ Pre-compiled on Debian Bullseye. Includes amd64 (x86_64) and arm64 builds starting v1.85.0.
Google Cloud Shell ✅ Native support via the Linux amd64 binary.
macOS (amd64 / arm64) ✅ Native support via the Mac binary. Includes amd64 (Intel) and arm64 (Apple Silicon) builds starting v1.85.0.
Windows ❎ Please Build from source.

[!NOTE] Multi-architecture builds (amd64 and arm64) are available starting with version 1.85.0. Tarball bundles (.tgz) are supported starting with version 1.89.0.

  1. Download and extract the bundle:

    For versions v1.89.0 and newer (Multi-architecture Tarball):

    bash
    # Find all available releases at: https://github.com/GoogleCloudPlatform/cluster-toolkit/releases
    # Set the desired version TAG (e.g., v1.89.0)
    TAG=vX.Y.Z
    # Set your OS (linux or mac) and architecture (amd64 or arm64)
    OS="linux"
    ARCH="amd64"
    # Download and extract the platform-specific bundle in a single step
    mkdir -p cluster-toolkit && curl -L https://github.com/GoogleCloudPlatform/cluster-toolkit/releases/download/${TAG}/gcluster_bundle_${OS}_${ARCH}.tgz | tar -xz -C cluster-toolkit && cd cluster-toolkit

    For versions v1.85.0 through v1.88.0 (Multi-architecture Zip):

    bash
    # Find all available releases at: https://github.com/GoogleCloudPlatform/cluster-toolkit/releases
    # Set the desired version TAG (e.g., v1.85.0)
    TAG=vX.Y.Z
    # Set your OS (linux or mac) and architecture (amd64 or arm64)
    OS="linux"
    ARCH="amd64"
    # Download and extract the platform-specific bundle
    curl -LO https://github.com/GoogleCloudPlatform/cluster-toolkit/releases/download/${TAG}/gcluster_bundle_${OS}_${ARCH}.zip
    unzip gcluster_bundle_${OS}_${ARCH}.zip -d cluster-toolkit/
    cd cluster-toolkit
  2. Verify the installation:

    bash
    ./gcluster --version
    ./gcluster --help

Building from source

If you prefer to build the gcluster binary from source, you can use the following commands:

bash
git clone https://github.com/GoogleCloudPlatform/cluster-toolkit
cd cluster-toolkit
make
./gcluster --version
./gcluster --help

[!NOTE] You must install dependencies (such as Go and Terraform) before building, otherwise the make command fails.

Prerequisites

Before deploying your first cluster, ensure the following are configured in your Google Cloud project.

Enable APIs

Several APIs must be enabled to deploy your cluster. While Terraform identifies missing APIs during terraform apply, enabling them upfront saves time. Required APIs typically include:

  • Compute Engine API
  • Filestore API
  • Cloud Storage API
  • Service Usage API

For more information, see Set up Cluster Toolkit.

Quotas

HPC and AI workloads often require significant resources. You might need to request additional quota to deploy your cluster. For more information, see Request additional quotas.

GCP credentials

Terraform can provide credentials for authenticating to Google Cloud in several ways. We recommend using gcloud on your workstation or using service accounts attached to cloud environments.

[!WARNING] We do not recommend downloading or using service account JSON keys. These keys are long-lived credentials that pose a significant security risk if leaked. Instead, use short-lived credentials via Application Default Credentials (ADC).

On your local terminal, Cloud Workstations, or Cloud Shell, generate Application Default Credentials (ADC) associated with your Google Cloud account:

bash
gcloud auth application-default login

Follow the prompts in your browser to authenticate. You are provided a token to copy and paste back into your terminal to complete the process. Once finished, Terraform automatically uses these "Application Default Credentials."

If you receive a quota project error, then set the quota project to your current project ID:

bash
gcloud auth application-default set-quota-project ${PROJECT_ID}

Telemetry and privacy notice

To help improve Cluster Toolkit, feature usage statistics are collected and sent to Google. You can opt-out at any time by executing the following command:

bash
./gcluster telemetry off

Cluster Toolkit telemetry overall is handled in accordance with the Google Privacy Policy. When you use Cluster Toolkit to interact with or utilize GCP Services, your information is handled in accordance with the Google Cloud Privacy Notice.

Cluster creation and management

After installing gcluster, you can deploy, manage, and destroy infrastructure using cluster blueprints:

  1. Create deployment folder: Use a blueprint YAML file to generate deployment files:

    bash
    ./gcluster create examples/hpc-slurm.yaml
  2. Deploy cluster: Provision infrastructure with Terraform:

    bash
    ./gcluster deploy hpc-slurm
  3. Destroy cluster: Clean up provisioned resources:

    bash
    ./gcluster destroy hpc-slurm

Automatic Dependency Management

When running commands like deploy or destroy in non-interactive environments (such as CI/CD pipelines), the toolkit can automatically download missing HashiCorp dependencies (Terraform and Packer) by passing the --download-dependencies flag.

  • --download-dependencies: Automatically download missing dependencies without prompting.
  • --download-dependencies=false: Fail immediately if any required dependencies are missing.

In interactive terminals, this flag is optional; the toolkit will ask for confirmation before downloading missing dependencies.

For detailed guides, blueprint syntax, and configuration options, see the Google Cloud documentation site.

Job submission (gcluster job submit)

The gcluster job submit command provides a unified interface to submit batch and distributed containerized workloads (JobSets) to your GKE clusters.

Prerequisites for job submission

Ensure your environment is set up before submitting jobs:

  • A deployed GKE cluster managed by Cluster Toolkit (with Kueue and JobSet enabled).
  • kubectl and gke-gcloud-auth-plugin installed and authenticated (gcloud auth login).
  • For on-the-fly builds (--build-context), set export GCLUSTER_IMAGE_REPO= to specify your Artifact Registry repository name (e.g., export GCLUSTER_IMAGE_REPO=gcluster-repo).

Configure default settings

To reduce boilerplate in job submit commands, you can set default configuration values for project, location, and cluster.

You can open your default system text editor interactively:

bash
./gcluster job config set

Or you can set values individually:

bash
./gcluster job config set project my-project
./gcluster job config set cluster my-cluster
./gcluster job config set location us-central1-a

Check the active configuration using:

bash
./gcluster job config show

Submit a JobSet

There are two ways to submit a JobSet with the gcluster job submit command:

  • Submit with a pre-built image: Submit a job using an existing container image from an image registry.
  • Submit with on-the-fly image building: Package local application code into a container image and push it automatically.

Submit with a pre-built image

To submit with a pre-built image, run ./gcluster job submit with the following properties in your shell:

bash
./gcluster job submit \
  --project  \
  --cluster  \
  --location  \
  --name my-job \
  --ima

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Go

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言