Bot defense: Disable unauthenticated access to media proxy

Pitch

Mastodon should have an admin option to disable unauthenticated access to the media proxy. To my understanding, unauthenticated requests that get punted to the media proxy currently only have lower rate limits than authenticated requests, which is ineffective against bot networks.

Motivation

Our instance is faced with a bot network that already seems to have indexed a large amount of valid article and media URLs by scraping our public timelines, tags search, and trends. (By now, we have disabled public access to remote / fediverse timelines and tags, but trends that include posts from remote instances remain an open avenue for indexing. Mastodon does not seem to have an option to disable trends access for unauthenticated users, only for everyone, but that's a different issue.)

In a previous iteration, this bot network has scraped public remote articles, triggering excessive media downloads into the cache (see https://github.com/mastodon/mastodon/issues/37359).

Right now, they seem to be working from a list of previously-stored URLs, and ends up requesting images through the media proxy on our instance, to the tune of 250.000 requests a day, originating from roughly 70.000 different source IPs. Every IP only makes a few request, avoiding rate limiting.

Since other forms of defense have been ineffective, we would like to be able to prevent unauthenticated users from accessing the media proxy altogether. Our current solution is to flat out block requests to /media_proxy/, which also makes the feature inaccessible to our own users.