Pull out bits of URLs provided on stdin
Pull out bits of URLs provided on stdin
If you have Go installed and configured:
▶ go install github.com/tomnomnom/unfurl@latestOtherwise download the latest binary for your platform,
extract it and move it to somewhere in your $PATH (e.g. /usr/bin/):
▶ wget https://github.com/tomnomnom/unfurl/releases/download/v0.0.1/unfurl-linux-amd64-0.0.1.tgz
▶ tar xzf unfurl-linux-amd64-0.0.1.tgz
▶ sudo mv unfurl /usr/bin/unfurl works with URLs provided on stdin; they might come from a file like this one:
▶ cat urls.txt
https://sub.example.com/users?id=123&name=Sam
https://sub.example.com/orgs?org=ExCo#about
http://example.net/about#contactYou can extract the domains from the URLs with the domains mode:
▶ cat urls.txt | unfurl domains
sub.example.com
sub.example.com
example.netIf you don't want to output duplicate values you can use the -u or --unique flag:
▶ cat urls.txt | unfurl --unique domains
sub.example.com
example.netThe -u/--unique flag works for all modes.
You can extract the apex part of the domain (e.g. the example.com in http://sub.example.com) using the apexes mode:
▶ cat urls.txt | unfurl -u apexes
example.com
example.net▶ cat urls.txt | unfurl paths
/users
/orgs
/about▶ cat urls.txt | unfurl keys
id
name
org▶ cat urls.txt | unfurl values
123
Sam
ExCo▶ cat urls.txt | unfurl keypairs
id=123
name=Sam
org=ExCo…You can use the format mode to specify a custom output format:
▶ cat urls.txt | unfurl format %d%p
sub.example.com/users
sub.example.com/orgs
example.net/aboutThe available format directives are:
…Any characters that don't match a format directive remain untouched:
▶ cat urls.txt | unfurl -u format "%d (%s)"
sub.example.com (https)
example.net (http)Note that if a URL does not include the data requested, there will be no output for that URL:
▶ echo http://example.com | unfurl format "%P"
▶ echo http://example.com:8080 | unfurl format "%P"
8080…No open issues yet, or sync has not completed.