url_for only returns URLs with an autrority component

(Re-raising from #604, since that issue was only partially fixed)

TL;DR: url_for should behave like it does in Flask and render absolute paths by default (unless an optional authority parameter is passed), not complete URLs.

---

Unlike Flask's url_for and Django's HttpRequest.build_absolute_uri(location), Starlette's Request.url_for always returns a complete URL with authority component. This tends to be unneccessarry when rendering templates that link to internal pages, but the important problem is that it also makes it so the app cannot render valid URLs unless it is aware of the authority the client is using to connect to it. Starlette uses the Host header to determine what authority to prepend to the generated URLs, which is a routing header and is spec-mandated to be overriden by any intermediate clients so that the application is only aware of what hostname the final request was directed towards. This means url_for is always going to break when Starlette is behind a proxy. The port and server values in the ASGI scope cannot differ from the Host header either, as they are meant to mirror what WSGI and CGI do.

Those are all meant to let the application reconstruct the URL of the request, but they can't be relied upon to build externally-valid URLs. Some proxies and application servers will send the X-Forwarded-Host or Forward headers to let the application know what authority the user agent was connecting from, but even the spec defining this header says

Due to the sensitive nature of the data passed in [Forwarded] (see Sections 8.2 and 8.3), this header field should be turned off by default.

Gunicorn removed the use of proxy headers several years ago when they realized overriding REMOTE_ADDR with the values from other headers was against the WSGI and CGI spec, because an application shouldn't have that sort of responsibility.

You simply can't dynamically generate externally valid URLs with an authority component. You either have to know the authority you want to use in advance from a config file (usually for internal redirection, like pointing towards a CDN) or just omit them and let the user agent understand that it is meant to be sent to the same server it got the rendered URL from.

[!IMPORTANT]

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.