Speed-up pushing to a fork by fetching remote heads with `git ls-remote`

Describe the issue

1. Host a repo with lfs objects on a remote server (gitlab / github)
2. Fork the repo
3. Clone the unforked repo locally
4. Locally create a branch and create a commit
5. Push the branch to the fork

Depending on repo size, this process takes a long time. We have a repo with around 300 000 commits and 350 000 lfs objects and it takes us around ~20 min.

As already analyzed in #4350, #3707 and #3915 this process takes a long time, because git lfs locally crawls through the history and then offers the lfs objects to the forked repo which it then denies.

However, if I fetch the master of the fork before pushing my local branch the push only takes a few ms.

Analysis I am not sure why this works, since I didn't have the time to look at the code of git or git lfs. I suspect it has something to do with the way the smart protocol communicates with the remote (https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols).

Question Is it possible to modify git lfs so that it always fetches the branches from the remotes before pushing? If this is not feasible, why not?

Thank you in advance for your time and thank you for creating git lfs in the first place :)