#8182·celery

Chaining a single group will not wait for group member completion

Author: EnteeeCreated Apr 6, 2023Updated Sep 15, 2026

LabelsComponent: CanvasIssue Type: Bug ReportStatus: Design Decision Needed ✘

Checklist

I have verified that the issue exists against the main branch of Celery.
This has already been asked to the discussions forum first. -> https://github.com/celery/celery/discussions/8170
I have read the relevant section in the contribution guide on reporting bugs.
I have checked the issues list for similar or identical bug reports.
I have checked the pull requests list for existing proposed fixes.
I have checked the commit log to find out if the bug was already fixed in the main branch.
I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue. (if you are not able to do this, then at least specify the Celery version affected).
I have verified that the issue exists against the main branch of Celery.
I have included the contents of pip freeze in the issue.
I have included all the versions of all the external dependencies required to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version and/or implementation.
I have tried reproducing the issue on more than one message broker and/or result backend.
I have tried reproducing the issue on more than one version of the message broker and/or result backend.
I have tried reproducing the issue on more than one operating system.
I have tried reproducing the issue on more than one workers pool.
I have tried reproducing the issue with autoscaling, retries, ETA/Countdown & rate limits disabled.
I have tried reproducing the issue after downgrading and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:5.3.0b2 (dawn-chorus) kombu:5.3.0b3 py:3.10.7
            billiard:4.1.0 py-amqp:5.1.1
platform -> system:Linux arch:64bit, ELF
            kernel version:5.15.94-1.qubes.fc32.x86_64 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:redis:///

broker_url: 'amqp://guest:********@localhost:5672//'
result_backend: 'redis:///'
deprecated_settings: None

Steps to Reproduce

Required Dependencies

Python Packages

pip freeze Output:

amqp==5.1.1
asttokens==2.2.1
async-timeout==4.0.2
backcall==0.2.0
billiard==4.1.0
celery @ git+https://github.com/celery/celery.git@2e168af493fb741e9ddc7261780a0070927351ed
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
decorator==5.1.1
executing==1.2.0
flower==1.2.0
humanize==4.6.0
ipdb==0.13.13
ipython==8.11.0
jedi==0.18.2
kombu==5.3.0b3
matplotlib-inline==0.1.6
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.16.0
prompt-toolkit==3.0.38
ptyprocess==0.7.0
pure-eval==0.2.2
Pygments==2.14.0
pytz==2022.7.1
redis==4.5.1
six==1.16.0
stack-data==0.6.2
tomli==2.0.1
tornado==6.2
traitlets==5.9.0
vine==5.0.0
watchdog==2.3.1
wcwidth==0.2.6

Other Dependencies

N/A

Minimally Reproducible Test Case

python

#!/usr/bin/env python3
from random import randint
from time import sleep
from celery import Celery, Signature, chain, chord, group
from celery.utils.log import  get_task_logger

app2 = Celery('app2', backend='redis://', broker='pyamqp://')
logger = get_task_logger(__name__)

@app2.task
def t1(*args):
    logger.info(f"{args[-1]}({args[:-1]})")
    return args

@app2.task
def t2(*args):
    logger.info(f"start {args[-1]}({args[:-1]})")
    sleep(randint(1, 5))
    logger.info(f"end {args[-1]}({args[:-1]})")
    return args

if __name__ == "__main__":
    chain(
        group(
            t2.s("group1"),
            chain(
                group(
                    t1.s("group2->chain1->group1"),
                    t1.s("group2->chain1->group2"),
                ),
            ),
        ),
        t1.s("chord"),
    ).delay()

Expected Behavior

The chord runs **after** the group1 task finishes:

[2023-03-31 10:59:39,365: INFO/ForkPoolWorker-2] Task app2.t2[592939f3-22bb-4e76-9b28-9a70b90a4f05] succeeded in 5.018022193999059s: ('group1',)
[2023-03-31 10:59:39,365: INFO/MainProcess] Task app2.t1[ad551c2a-e7a0-4191-9eea-983b83f99ada] received
[2023-03-31 10:59:39,366: INFO/ForkPoolWorker-2] app2.t1[ad551c2a-e7a0-4191-9eea-983b83f99ada]: chord(([['group1'], [['group2->chain1->group1'], ['group2->chain1->group2']]],))
[2023-03-31 10:59:39,367: INFO/ForkPoolWorker-2] Task app2.t1[ad551c2a-e7a0-4191-9eea-983b83f99ada] succeeded in 0.0007412640006805304s: ([['group1'], [[...], [...]]], 'chord')

Actual Behavior

The chord runs before the group1 task finishes:

[2023-03-31 10:40:25,661: INFO/MainProcess] Task app2.t1[4e447663-d92e-46fe-923a-3199b353df28] received
[2023-03-31 10:40:25,662: INFO/ForkPoolWorker-3] app2.t1[4e447663-d92e-46fe-923a-3199b353df28]: chord(([None, [['group2->chain1->group1'], ['group2->chain1->group2']]],))
[2023-03-31 10:40:25,663: INFO/ForkPoolWorker-3] Task app2.t1[4e447663-d92e-46fe-923a-3199b353df28] succeeded in 0.0013457320019369945s: ([None, [[...], [...]]], 'chord')
[2023-03-31 10:40:27,098: INFO/ForkPoolWorker-2] app2.t2[89265d0d-a876-4ce8-9e1d-d7f68a38246d]: end group1(())
[2023-03-31 10:40:27,101: INFO/ForkPoolWorker-2] Task app2.t2[89265d0d-a876-4ce8-9e1d-d7f68a38246d] succeeded in 2.0062880830009817s: ('group1',)

Source: celery/celery

View original on GitHub View discussion on GitHub