百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
H

hyperqueue

> 编程语言
开源

面向 HPC 系统的用户友好子节点任务调度器

549 stars0 点赞2 次浏览
访问官网GitHub

工具介绍

面向 HPC 系统的用户友好子节点任务调度器

HyperQueue is a tool designed to simplify execution of large workflows (task graphs) on HPC clusters. It allows you to execute a large number of tasks in a simple way, without having to manually submit jobs into batch schedulers like Slurm or PBS. You specify what you want to compute and HyperQueue automatically asks for computational resources and dynamically load-balances tasks across all allocated nodes and resources. HyperQueue can also work without Slurm/PBS as a general distributed task execution engine.

Documentation

  • You can check out examples to see how to use HyperQueue.
  • If you find a bug or a problem with HyperQueue, please create an issue.
  • For more general discussion or feature requests, please use our discussion forum.
  • If you want to chat with the HyperQueue developers, you can use our Zulip server.

If you use HyperQueue in your research, please consider citing it.

This image shows how HyperQueue can work on a distributed cluster that uses Slurm or PBS:

You can find a talk from the e-INFRA conference that describes the basics of HyperQueue here.

Demo

This demo shows simple usage of HyperQueue to execute a few hundred commands (tasks), with dynamic scaling both up and down based on the number of connected workers.

Features

  • Complex resource management

    • Load balancing of tasks across all available (HPC) resources
    • Automatic submission of Slurm/PBS jobs on behalf of the user
    • Complex and arbitrary task resource requirements (# of cores, GPUs, memory, FPGAs, ...)
      • Non-fungible resources (tasks are assigned specific resources, e.g. a GPU with ID 1)
      • Fractional resources (tasks can require e.g. 0.5 of a GPU)
      • Resource variants (tasks can require e.g. 1 GPU and 4 CPU cores OR 16 CPU cores)
      • Related resources (tasks can require e.g. 4 CPU cores in the same NUMA node)
  • High performance

    • Scales to hundreds of nodes/workers and millions of tasks
    • Overhead per one task is below 0.1ms
    • Allows streaming of stdout/stderr from tasks to avoid creating many small files on distributed filesystems
  • Simple user interface

    • Task graphs can be defined via a CLI, TOML workflow files or a Python API
    • Cluster utilization can be monitored with a real-time dashboard
  • Easy deployment

    • Provided as a single, statically linked binary without any runtime dependencies (apart from libc)
    • No admin access to a cluster is needed for its usage

Installation

  • Download the latest binary distribution from this link.

  • Unpack the downloaded archive:

    bash
    $ tar -xvzf hq--linux-x64.tar.gz
  • That's it! Just use the unpacked hq binary.

If you want to try the newest features, you can also download a nightly build.

Submitting a simple task

  • Start a server (e.g. on a login node, a cluster partition, or simply on your PC)

    bash
    $ hq server start &
  • Submit a job (command echo 'Hello world' in this case)

    bash
    $ hq submit echo 'Hello world'
  • Ask for computing resources

    • Either start a worker manually

      bash
      $ hq worker start &
    • Or configure automatic submission of workers into PBS/SLURM

      • PBS:

        bash
        $ hq alloc add pbs --time-limit 1h -- -q 
      • Slurm:

        bash
        $ hq alloc add slurm --time-limit 1h -- -p 
  • See the result of the job once it finishes

    bash
    $ hq job wait last
    $ hq job cat last stdout

What's next?

Check out the documentation.

You can find FAQ (frequently asked questions) here.

HyperQueue team

We are a group of researchers working at IT4Innovations, the Czech National Supercomputing Center. We welcome any outside contributions.

Publications

  • HyperQueue: Efficient and ergonomic task graphs on HPC clusters (paper @ SoftwareX journal)
  • HyperQueue: Overcoming Limitations of HPC Job Managers (poster @ SuperComputing'21)

If you want to cite HyperQueue, you can use the following BibTex entry:

bibtex
@article{hyperqueue,
  title = {HyperQueue: Efficient and ergonomic task graphs on HPC clusters},
  journal = {SoftwareX},
  volume = {27},
  pages = {101814},
  year = {2024},
  issn = {2352-7110},
  doi = {https://doi.org/10.1016/j.softx.2024.101814},
  url = {https://www.sciencedirect.com/science/article/pii/S2352711024001857},
  author = {Jakub Beránek and Ada Böhm and Gianluca Palermo and Jan Martinovič and Branislav Jansík},
  keywords = {Distributed computing, Task scheduling, High performance computing, Job manager}}
}

Acknowledgement

  • This work was supported by the LIGATE project. This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 956137. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Italy, Sweden, Austria, the Czech Republic, Switzerland.

  • This work was supported by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90140).

License

MIT

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Rustdistributed-computinghpcrusttask-graph

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言