GitLab CI_JOB_TOKEN + internal package does not work - but works with `source-fallback`

TL;DR

when using a internal/private GitLab repository as a dependency, CI_JOB_TOKEN does not currently work as a mean of authentication inside a GitLab CI pipeline; source-fallbackmakes it work anyway, hiding the issue.

Having source-fallbackgo away before fixing this would break this process, forcing us to ditch a very secure approach (by either pinning to Composer 2.10, or by ditching CI_JOB_TOKEN).

Issue details

Why CI_JOB_TOKEN?

CI_JOB_TOKEN is a very secure mean of authentication inside a GitLab CI pipeline, because:

• it's ephemeral: the token is valid only until the job ends
• it doesn't require manual setup or rotation
• it's not bound to a specific user: it impersonates the user that triggered the job execution
• it has additional security limitations:
  • is authorized to do only certain operations
  • scope can be further reduced in terms of which repositories and which actions

See docs: https://docs.gitlab.com/ci/jobs/ci_job_token/

Reproduction

Setup:

• A GitLab PHP project, which depends on a GitLab internal/private PHP dependency (my/private-package)
• composer install run in GitLab CI for the project
• usage of CI_JOB_TOKEN in CI to authenticate against GitLab itself

{
    "require": {
        "my/private-package": "^1.0"
    },
    "config": {
        "gitlab-domains": [
            "gitlab.example.com"
        ],
    }
}

Pipeline execution:

composer config http-basic.gitlab.example.com -- gitlab-ci-token "$CI_JOB_TOKEN"
composer install

Before 2.10, or with source-fallback set to true:

Failed to download my/private-package from dist: Could not authenticate against gitlab.facile.it
Now trying to download from source
 - Syncing my/private-package (v1.0.0)

...and it works.

With source-fallback not set in 2.10 or set to false :

Failed to download my/private-package from dist: Could not authenticate against gitlab.example.com
Source fallback is disabled. Not trying alternative sources.


In AuthHelper.php line 169:

Could not authenticate against gitlab.example.com

What should be done

I've already opened #12916 to fix Composer's behavior when handling CI_JOB_TOKEN, because it was sent as a PRIVATE-TOKEN instead of a correct JOB-TOKEN header. This is unfortunately not enough, because while testing my PR I discovered that the token doesn't have access to the needed API by design. I've opened an issue on GitLab to fix this too: https://gitlab.com/gitlab-org/gitlab/-/work_items/602168