百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
P

postrank-uri

> 编程语言
开源

URI 正则化、c14n、转义和提取

301 stars0 点赞1 次浏览
访问官网GitHub

工具介绍

URI 正则化、c14n、转义和提取

PostRank URI

A collection of convenience methods (Ruby 2.3+) for dealing with extracting, (un)escaping, normalization, and canonicalization of URIs. At PostRank we process over 20M URI associated activities each day, and we need to make sure that we can reliably extract the URIs from a variety of text formats, deal with all the numerous and creative ways users like to escape and unescape their URIs, normalize the resulting URIs, and finally apply a set of custom canonicalization rules to make sure that we can cross-reference when the users are talking about the same URL.

In a nutshell, we need to make sure that creative cases like the ones below all resolve to same URI:

  • http://igvita.com/
  • http://igvita.com///
  • http://igvita.com/../?#
  • http://igvita.com/a/../?
  • http://igvita.com/a/../?utm_source%3Danalytics
  • ... and the list goes on - check the specs.

API

  • PostRank::URI.extract(text) - Detect URIs in text, discard bad TLD's

  • PostRank::URI.clean(uri) - Unescape, normalize, apply c14n filters - 95% use case.

  • PostRank::URI.normalize(uri) - Apply RFC normalization rules, discard extra path characters, drop anchors

  • PostRank::URI.unescape(uri) - Unescape URI entities, handle +/%20's, etc

  • PostRank::URI.escape(uri) - Escape URI

Example

>> PostRank::URI.extract('some random text with http://link.to somecanadiansite.ca')
[
    [0] "http://link.to/",
    [1] "http://somecanadiansite.ca/"
]

>> PostRank::URI.clean('link.to?a=b&utm_source=FeedBurner#stuff')
[
    [0] "http://link.to/?a=b"
]

C14N

As part of URI canonicalization the library will remove common tracking parameters from Google Analytics and several other providers. Beyond that, host-specific rules are also applied. For example, nytimes.com likes to add a 'partner' query parameter for tracking purposes, but which has no effect on the content - hence, it is removed from the URI. For full list, see the c14n.yml file.

Detecting "duplicate URLs" is a hard problem to solve (expensive in all senses), instead we are compiling a manually assembled database. If you find cases which are missing, please do report them, or send us a pull request!

Development

Setup

bundle install

Running tests

bundle exec rake

Running dependency appraisals

To verify postrake-uri works with different versions of its runtime dependencies you can run:

bundle exec appraisal install
bundle exec rake appraisal

This will execute the test suite with different versions of the dependencies.

Issues· 3 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Ruby

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言