[Bug] Webhook deploys fail permanently when .git/rebase-merge is left in Core's repo cache

Webhook deploys fail permanently after an interrupted rebase leaves .git/rebase-merge in Core's repo cache

What happened

We push to GitLab, a webhook fires, and Komodo redeploys. One day the redeploy stopped happening. The push went through, the webhook fired, and the deploy failed on a git error.

Nobody noticed at first. Webhook-driven deploys are unattended by design, so the failure was silent. It only surfaced when someone wondered why a change had not landed. By then the deployment had been stale for several days, and in the meantime nothing could be updated or changed at all — every deploy hit the same error.

The error was this, identically on every attempt:

warning: redirecting to https://gitlab.example.com/[GROUP]/[REPO].git/
From https://gitlab.example.com/[GROUP]/[REPO]
* branch            main       -> FETCH_HEAD
fatal: It seems that there is already a rebase-merge directory, and
I wonder if you are in the middle of another rebase.  If that is the
case, please try
  git rebase (--continue | --abort | --skip)
If that is not the case, please
  rm -fr ".git/rebase-merge"
and run me again.  I am stopping in case you still have something
valuable there.

(Remote URL redacted; it is a private GitLab repo.)

What we found

The error message sent us to the wrong place first. We checked the clone on the server: clean working tree, no conflict markers anywhere, main exactly at origin/main, and no sign of a rebase in progress. git status reported nothing unusual. So the error made no sense against what we were looking at.

The clone that was actually broken was the one in Komodo Core's repo cache. That clone was stuck mid-rebase, with .git/rebase-merge still present from some earlier interrupted run. Because it is Core's cache, the impact was not one failing deploy on one server — everything resolving through that clone was stuck.

It never recovered on its own. Retrying, redeploying and further webhook pushes all hit the same error. The only thing that fixed it was re-initialising the cached clone from scratch.

Why it gets into this state and stays there

git pull --rebase --force origin <branch> is run against the cached clone (stage order per lib/git/src/pull.rs, as documented in #1630). If that rebase is ever interrupted — a conflict, a service restart, a timeout, or two operations touching the same clone concurrently — git leaves .git/rebase-merge behind by design, on the assumption that a human will come back and resolve it.

Nothing in the pull path cleans it up. There is no pre-flight check for .git/rebase-merge or .git/rebase-apply, and no git rebase --abort in the error path. So a one-off interruption becomes a permanent failure for every subsequent pull on that clone.

Worth noting that the cached clone can diverge from the remote without anyone touching it, so this is reachable in a normal hands-off setup. Once it has diverged, the next upstream commit turns what would have been a fast-forward into an actual rebase — which is where it can stop.

We could not determine what interrupted the rebase originally, because the clone was already gone by the time we went looking. One plausible route: redeploys are webhook-driven and polling runs daily, so two pulls can overlap on the same cached clone.

Steps to reproduce

1. Configure a Repo resource against a git remote, with poll_for_updates enabled and push webhooks triggering redeploys.
2. Get Core's cached clone into a state where the local branch has diverged from origin/<branch> (a local commit in the cache, or a rewritten upstream history).
3. Trigger a pull while the remote has moved on, so git pull --rebase performs an actual rebase, and interrupt it (conflicting change, or restart the Core service mid-pull).
4. Run PullRepo again — and every time after that.

Expected: Komodo recovers, or at least reports that the clone is in an interrupted-rebase state and offers to reset it.

Actual: every pull fails with the error above, indefinitely, for everything sharing that clone.

Suggested fix

• Before pulling, check for .git/rebase-merge and .git/rebase-apply in the cached clone. If present, run git rebase --abort (or remove the directory) and continue. In a machine-managed cache there is nothing worth preserving, so git's "I am stopping in case you still have something valuable there" reasoning does not apply — but it is exactly that reasoning that turns a transient failure into a permanent one here.
• Add git rebase --abort to the pull error path, so a failed pull leaves the clone usable for the next run.
• Offer a hard-reset pull strategy (git fetch && git reset --hard origin/<branch>) as an alternative to the rebase pull. Already requested in #553, and it would avoid this whole class of problem — a deploy clone arguably should never rebase. Related: #501, where --force in this path cost a user commits.
• Make the failure diagnosable. A failing pull currently reports git's raw stderr and little else: no indication of which clone was used, what state it was in, or that the failure is permanent rather than transient. Logging the resolved clone path and a distinct error for "clone is in an interrupted-rebase state" would have pointed us at Core's cache immediately instead of at the server clones. Related, #1630 notes that the explicit fetch's log entry is only pushed to the response on failure, so on the success path the stage is invisible.
• If concurrent pulls on the same cached clone are possible, serialising them per clone would remove one way of getting into this state.

On the silent-failure part: we did not have alerting enabled, so this is not a report that notifications failed to fire. But since webhook-triggered deploys are unattended by default, it may be worth considering whether a repeatedly failing deploy on the same resource should be surfaced more prominently than inside the update view.

Environment

• Komodo Core version (GetVersion): 2.3.2
• Komodo Periphery version: 2.3.2
• Install method: systemd
• OS / arch: Ubuntu 24.04, x86_64
• git version on the Core host: 2.43.0
• Affected clone: Komodo Core's repo cache
• Resource type: Repo
• Repo remote: GitLab over HTTPS
• poll_for_updates: enabled, once per day. Redeploys additionally triggered by GitLab push webhooks.
• Duration before the stale deployment was noticed: several days at minimum

Related

• #1630 — documents the PullRepo stage order including git pull --rebase --force origin <branch>
• #553 — request for a git hard reset option
• #501 — data loss via --force in the same path
• #1629 — PullRepo update state gated on RefreshRepoCache