Parallel Workspace Applies Fail since 0.46.0

Community Note

• Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Since v0.46.0 (specifically #6605), pull-level apply lock are enforced for every apply, targeted or not. Two atlantis apply -w <workspace> comments on the same PR can no longer run in parallel:

cannot run "apply": pull request 22 is currently locked by "apply" for commit 12345678.
Wait until the previous command is complete and try again

On previous versions locks look to have been keyed by repo/pull/workspace/path/project. That means targeted applies for different workspaces were permitted.

Our workflow is to have one repo with multiple workspaces each representing an enviornment/cluster. In many cases we want to apply multiple at once (daily-1, daily-2) while waiting to apply additional environments (integration-1, integration-2).

We use one command per workspace because we want each environment to have its own comment and job link and to be independently re-runnable when one fails. To that end we also set --disable-apply-all to avoid someone applying all clusters at once without an initial smoke deploy.

Reproduction Steps

1. A repo with a single root module and two or more workspaces, and an atlantis.yaml with parallel_apply: true and one project per workspace all sharing dir: ./.
2. Push a commit that touches a shared .tf so autoplan plans every workspace. Let it finish.
3. Comment atlantis apply -w daily-1.
4. Within a few seconds, comment atlantis apply -w daily-2.
5. The second command is rejected with the error above. On v0.45.0 both applies run concurrently.

Logs

cannot run "apply": pull request 22 is currently locked by "apply" for commit a1b2c3d. Wait until the previous command is complete and try again

The first atlantis apply -w daily-1 was commented at 13:50:17Z and completed at 13:58:07Z. The second, atlantis apply -w daily-2, was commented at 13:50:23Z and rejected at 13:50:27Z.

Environment details

• Atlantis version: v0.47.1
• Deployment method: Kubernetes StatefulSet running the official ghcr.io/runatlantis/atlantis image
• Atlantis flags: --disable-apply-all=true, --parallel-pool-size=25, --automerge=true, --default-tf-distribution=opentofu, --gh-allow-mergeable-bypass-apply=true

Sever-side repo config:

Atlantis server-side config file (plan steps elided):

repos:
  - id: /.*/
    branch: /(main|master)/
    apply_requirements: [mergeable]
    plan_requirements: [undiverged]
    pre_workflow_hooks:
      - run: sh ./generate-atlantis-yaml.sh && cat ./atlantis.yaml
        description: Generate dynamic atlantis.yaml
        commands: plan
workflows:
  default:
    apply:
      steps:
        - run: tofu workspace select -no-color $WORKSPACE
        - run: tofu apply -parallelism=100 -no-color $PLANFILE 2>&1 | tee ./apply.out
        - run: sh ./rollout-check.sh
        - run: sh ./annotate-grafana.sh

Repo atlantis.yaml

version: 3
parallel_plan: true
parallel_apply: true
projects:
  - dir: ./
    autoplan:
      when_modified: ["./**/*","!./**/*md","!./*tfvars","./daily-1.tfvars"]
    workspace: daily-1
  - dir: ./
    autoplan:
      when_modified: ["./**/*","!./**/*md","!./*tfvars","./daily-2.tfvars"]
    workspace: daily-2
  - dir: ./
    autoplan:
      when_modified: ["./**/*","!./**/*md","!./*tfvars","./integration-1.tfvars"]
    workspace: integration-1
  - dir: ./
    autoplan:
      when_modified: ["./**/*","!./**/*md","!./*tfvars","./integration-2.tfvars"]
    workspace: integration-2

Additional Context

• #6786 appears to hit the same lock in a slightly different way: multi-instance -p targeting and --silence-no-projects.
• Our apply workflow is custom run steps with no built-in apply step, because we need tofu workspace select, a rollout check and a Grafana annotation around the apply. The Custom Workflow Boundary section of #6605 says those steps are outside Atlantis-managed apply-step validation, so we are serialized by a lock whose purpose is protection we dont get. #6781 carved custom workflows out of the managed-plan-file requirement for the same reason.