Deduplicate crawled URLs before template execution

Each input line is stored as its own target, and only exact duplicates are removed. {{BaseURL}} keeps the path, so two URLs on the same host run every template twice.

Normalize the URL first: lowercase the host, drop the fragment and default port, trim a trailing slash, and remove tracking parameters. Templates whose path is {{RootURL}}, or {{BaseURL}} plus a fixed suffix, run once per origin. Templates that request the crawled URL keep one representative per structural fingerprint: replace numeric IDs, UUIDs, hashes, and dates in the path, collapse a path position once it has many distinct values on that host, and keep sorted query names with the values dropped. If many patterns on one host return the same body, stop adding patterns for that host.