百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
U

urless

> 编程语言
开源

清理 URL 列表

391 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

清理 URL 列表

About - v2.7

This is a tool used to de-clutter a list of URLs. As a starting point, I took the amazing tool uro by Somdev Sangwan. But I wanted to change a few things, make some improvements (like deal with GUIDs) and make it more customizable.

Installation

urless supports Python 3.

Install urless in default (global) python environment.

bash
pip install urless

OR

bash
pip install git+https://github.com/xnl-h4ck3r/urless.git -v

You can upgrade with

bash
pip install --upgrade urless

pipx

Quick setup in isolated python environment using pipx

bash
pipx install git+https://github.com/xnl-h4ck3r/urless.git

Usage

Argument Long Argument Description
-i --input A file of URLs to de-clutter.
-o --output The output file that will contain the de-cluttered list of URLs (default: output.txt). If piped to another program, output will be written to STDOUT instead.
-fk --filter-keywords A comma separated list of keywords to exclude links (if there no parameters). This will override the FILTER_KEYWORDS list specified in config.yml
-fe --filter-extensions A comma separated list of file extensions to exclude. This will override the FILTER_EXTENSIONS list specified in config.yml
-rp --remove-params A comma separated list of case senistive parameters to remove from ALL URLs. This will override the REMOVE_PARAMS list specified in config.yml. This can be useful to remove cache buster parameters for example.**
-ks --keep-slash A trailing slash at the end of a URL in input will not be removed. Therefore there may be identical URLs output, one with and one without a trailing slash.
-khw --keep-human-written By default, any URL with a path part that contains more than 3 dashes (-) are removed because it is assumed to be human written content (e.g. blog post), and not interesting. Passing this argument will keep them in the output.
-kym --keep-yyyymm By default, any URL with a path containing /YYYY/MM (where YYYY is a year and MM month) are removed because it is assumed to be blog/news content, and not interesting. Passing this argument will keep them in the output.
-rcid --regex-custom-id USE WITH CAUTION! Regex for a Custom ID that your target uses. Ensure the value is passed in quotes. See the section below for more details on this.
-iq --ignore-querystring Remove the query string (including URL fragments #) so output is unique paths only.
-fnp --fragment-not-param Don't treat URL fragments # in the same way as parameters, e.g. if a link has a filter keyword and a fragment (or param) the link is usually kept, but if this argument is passed and a link has a filter word and fragment, the link will be removed. Also, if this arg is passed and -iq / --ignore-querystring is used, the fragment will NOT be removed from links if no query string is in the link.
-lang --language If passed and there are multiple URLs with different language codes as a part of the path, only one version of the URL will be output. The codes are specified in the LANGUAGE section of config.yml.
-c --config Path to the YML config file. If not passed, it looks for file config.yml in the default config directory, e.g. ~/.config/urless/.
-dp --disregard-params There is certain filtering that is not done if the URLs have parameters, because by default we want to see all possible parameters. If this argument is passed, then the filtering will be done, regardless of the existence of any parameters.
-nb --no-banner Hides the tool banner (it is hidden by default if you pipe input to urless) output.
--version Show current version number.
-v --verbose Verbose output

What does it do exactly?

You basically pass a list of URLs in (from a file, or pipe from STDIN), and get a de-cluttered file or URLs out. But in what way are they de-cluttered? I'll explain this below, but first here are some terms that will be used:

  • FILTER-EXTENSIONS: This refers to the list of extensions that can either be passed with -fe, specified with FILTER_EXTENSIONS in the config.yml, or if neither of those exist, a default list of .css,.ico,.jpg,.jpeg,.png,.bmp,.svg,.img,.gif,.mp4,.flv,.ogv,.webm,.webp,.mov,.mp3,.m4a,.m4p,.scss,.tif,.tiff,.ttf,.otf,.woff,.woff2,.bmp,.ico,.eot,.htc,.rtf,.swf,.image.
  • FILTER-KEYWORDS: This refers to the list of keywords that can either be passed with -fk, specified with FILTER_KEYWORDS in the config.yml, or if neither of those exist, a default list of blog,article,news,bootstrap,jquery,captcha,node_modules
  • LANGUAGE: This refers to the list of language codes that can be specified with LANGUAGE in the config.yml, or if it doesn't exist, a default list of the most common codes en,en-us,en-gb,fr,de,pl,nl,fi,sv,it,es,pt,ru,pt-br,es-mx,zh-tw,js.ko
  • UNWANTED-CONTENT:
    • A section of the URL path contains more than 3 dashes (-), BUT isn't a GUID. This implies human written content, e.g. how-to-hack-the-planet. If arg -khw is passed, then this won't be removed.
    • The URL contains /YYYY/MM/ , e.g. a year, month . This is usually static content such as a blog. If arg -kym is passed, then this won't be removed.

Here's what happens:

  • If a URL has port 80 or 443 explicitly given, then remove it from the URL (e.g. http://example.com:80/test -> http://example.com/test)
  • If the URL has any FILTER-EXTENSIONS, it will be removed from the output.
  • If the URL has NO parameters OR the -dp/--disregard-params argument was passed:
    • If the URL contains a FILTER-KEYWORDS or UNWANTED-CONTENT, it will be removed.
    • if the URL query string contains unwanted parameters specified in config REMOVE_PARAMS (or overridden wit argument -rp/--remove-params), they will be removed from all URLs before processing.
    • If -rcid/--regex-custom-id is passed and the URL path contains a Custom ID, only one match to the Custom ID regex will be included if there are multiple URLs where that is the only difference.
    • If the URL path contains a GUID, only one of the GUIDs will be included if there are multiple URLs where the GUID is the only difference.
    • If the URL path contains an Integer ID, only one of the Integer IDs will be included if there are multiple URLs where the Integer ID is the only difference.
    • If the -lang argument is passed and the URL contains a language code (e.g. en-gb), only one of the language codes will be included if there are multiple URLs where the language code is different.
  • Else the URL has Parameters (or a fragment #) AND the -dp/`--disrega

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言