GitLab: a third-party commit-status poster strands Atlantis's own external pipeline in `running` forever

Community Note

• Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

We run Atlantis on a GitLab project where a second integration (CodeRabbit in our case) also posts commit statuses. Whenever both post on the same commit, Atlantis creates an external pipeline for its first status write and then abandons it: the pipeline sits at running forever, even after the MR is merged. When we went looking, we found seven of these orphans on one repo, the oldest from May 2024: this had been happening quietly for over a year before anyone noticed.

The mechanism is in UpdateStatus in server/events/vcs/gitlab/client.go (same code at v0.47.1 and on current main):

1. It fetches the commit, and if commit.LastPipeline is set, posts the status with PipelineID = commit.LastPipeline.ID.
2. Otherwise it retries the lookup a couple of times, then falls back to Ref = pull.HeadBranch, the short branch name, which makes GitLab find-or-create a pipeline.

Nothing checks that commit.LastPipeline is the pipeline Atlantis itself created. So if another integration creates a newer pipeline on the same commit between Atlantis's first and second writes, Atlantis silently switches to it, and the pipeline it created never reaches a terminal state.

The reason there are two pipelines at all: GitLab's set-commit-status API is find-or-create keyed on the ref string, and the two posters spell the same branch differently: Atlantis's fallback uses the bare branch name (<branch>), CodeRabbit uses the fully-qualified refs/heads/<branch>. GitLab treats those as distinct refs and creates a pipeline for each.

Reproduction Steps

1. A GitLab project with no CI jobs of its own, so no MR pipeline exists and Atlantis always takes the Ref = pull.HeadBranch fallback for its first write.
2. A second integration that posts commit statuses for the same commits using the fully-qualified refs/heads/<branch> ref.
3. Open an MR. Atlantis autoplans and posts atlantis/plan = running first, creating pipeline A keyed on the short ref.
4. The second integration posts its status a few seconds later, creating pipeline B keyed on refs/heads/<branch>.
5. Atlantis's subsequent writes find commit.LastPipeline = B and post there.
6. Pipeline A stays running forever, including after the MR is merged.

For one incident we have the full timeline: Atlantis created pipeline A at 13:10:36.517Z and wrote to it exactly once, 141 ms later. CodeRabbit created pipeline B at 13:10:40.189Z, 3.7 s after Atlantis gave up looking, and B then received every one of Atlantis's remaining statuses, ending success at 13:20:16.860Z. Plan and apply both succeeded and the MR merged normally; only the pipeline record is wrong.

We could also watch the adoption happen live after enabling gitlab-status-retry-enabled (see Logs below): Atlantis retries, then logs that it found a pipeline and adopts it, but the pipeline it adopts was created by CodeRabbit, not by Atlantis.

Logs

The bug itself produces no errors: everything succeeds from Atlantis's point of view, so there is nothing to show for the failure case. What we can show is the adoption decision. With gitlab-status-retry-enabled: true, on a commit where CodeRabbit's pipeline appeared while Atlantis was still retrying:

No pipeline found for commit e19727f35c9d1265febf6e601644c37a00dec2be, retrying in 2s
No pipeline found for commit e19727f35c9d1265febf6e601644c37a00dec2be, retrying in 3.4194057s
Pipeline found for commit e19727f35c9d1265febf6e601644c37a00dec2be, setting pipeline ID to 237707

Pipeline 237707 was created by the other integration. Atlantis attaches all its statuses to it by ID from then on. In this instance that happens to be harmless (one pipeline, one poster driving it to terminal state), but it is the same unchecked adoption that strands Atlantis's own pipeline in the race described above.

One more practical consequence for anyone else cleaning these up: the stranded pipelines contain no jobs, so GitLab's stuck/timeout handling never touches them, and POST /pipelines/:id/cancel returns 200 but does not change their status. The only way we found to clear them was DELETE /pipelines/:id as Owner.

Environment details

• Atlantis v0.47.1 (ghcr.io/runatlantis/atlantis:v0.47.1-debian), single instance on ECS Fargate, EFS-backed data dir
• GitLab self-managed 19.3.1
• Terragrunt workflow; atlantis.yaml generated per run by a pre-workflow hook; --parallel=false
• gitlab-status-retry-enabled was false (the default) when the orphans were created; we have since enabled it

Additional Context

• This looks like the same underlying defect as #5228 (still open): both cases come from adopting commit.LastPipeline.ID without checking that the pipeline belongs to this branch or to Atlantis. #5228's trigger is two MRs sharing a HEAD commit; ours is a third-party status poster, which we didn't find covered in any existing issue.
• #5944 / PR #5986 (the gitlab-status-retry-enabled flag) helps but doesn't close the hole: the longer wait makes it likely Atlantis finds the other pipeline before ever creating its own, so the window is narrowed, not removed, and when it does find one, it still adopts it without checking whose it is.
• A durable fix would be to check the ref and/or owner of commit.LastPipeline before adopting it, or to have Atlantis remember and reuse the pipeline it created for a given commit.

Happy to test a patch against a live GitLab 19.3.1 setup.