Drafting a new specification of radar rules
Context
#635
Motivation
Currently, the target field can be either 1. a route string; or 2. a function. The second use case could potentially lead to some security worry. What's more, it effectively makes radar rules unable to be converted to a JSON but remains to be a Javascript. That is, to load remote or local-modified rules cannot avoid the usage of eval(), which is strictly prohibited by some modern browsers (e.g. Firefox and Safari, total market share is ~18%) by default. We have already seen there are two distributions (AMO, Mac App Store) that have to disable remote rules and ignore local modifications. And no one can promise that Chrome Web Store would not ban the usage of eval() in the future too.
Another reason is most radar rules use target function never need to access the page content, but just match URL query strings / do some filtering/remapping/regex replacement. These rules should have been compatible with RSSBud/RSSAid, but with the current specification, they can only use a target function which is incompatible with RSSBud/RSSAid. The new specification should change these embarrassing situations.
Last but not least, some users indeed have their demand to use their own online rule list instead of the official one. Though the user is the only one who is responsible for their action, we should still worry about their data security since using eval() to load remote rules is typical remote code execution (RCE). As a result, such a feature could not be added before we completely deprecated the usage of eval(), and the prerequisite is this issue.
Core Idea
A typical target function can be described in these three workflows or there combination:
- 1st positional parameter
param-> post-process (string operations) -> filter -> remap - 2nd positional parameter
URL-> extract qs -> ... - 3rd positional parameter
document-> DOM operations -> ...
Thus, I constructed a 4-stage workflow:
- match -> post-process -> filter -> remap
But after deep diving, I think there's no need to distinguish these stages, we just need to define some actions and let rule authors chain them:
matcher: {
paramName: [
{
action: ...,
...: ...,
},
{...},
{...},
...
]
}Actions
Universal parameters
action: action namestopIfEmpty: whether to stop the chain immediately if the output of the current action is an empty string? (optional, default:true)
Input (must be placed at the beginning of the chain)
- URL path:
{ action: 'path' },{ action: 'path', key: '...' }(formerlyparam[paramName]) - URL qs:
{ action: 'qs' },{ action: 'qs', key: '...' } - raw URL:
{ action: 'URL' } - raw document as a string:
{ action: 'raw' } querySelector:{ action: 'DOM', ...(To be determined) }(maybe we could implement some workarounds to make it chainable?)- More? Share your ideas ☺️
Modify
- regex matching:
{ action: 'regex', match: '...' },{ action: 'regex', match: '...', matchGroup: 1 } - regex replacement:
{ action: 'regex', match: '...', replace: '...' } string operation(regex matching/replacement should be enough)- More? Share your ideas ☺️
Filter
Hmm, regex replacement should be enough, but that's OK to provide a convenient action.
Remap
{
action: 'remap',
map: {
key1: 'value1',
key2: { value: 'value2', overrideTitle: 'title2' }
}
default: 'defaultValue', // optional, default: ''
}A Quick Look at the New Specification
Need more discussion to determine more details.
Due to the support for optional parameters, #693 is a prerequisite of the new specification.
Under the new specification, the radar rule list should be exported both in .js and .json. The former, which contains all fields, is preserved for backward compatibility; the latter, in which target functions are filtered out and old target route strings are remapped to route, will be used by new versions of RSSHub Radar.
{
'12306.cn': {
_name: '12306',
kyfw: [
{
title: '售票信息',
docs: 'https://docs.rsshub.app/travel.html#_12306',
source: ['/', '/otn/leftTicket/init'],
target: (params, url) => { // for backward compatibility, will be dropped in JSON output
const searchParams = new URL(url).searchParams;
const from = searchParams.get('fs').split(',')[0];
const to = searchParams.get('ts').split(',')[0];
const date = searchParams.get('date');
return `/12306/${date}/${from}/${to}`;
},
route: '/12306/:date/:from/:to/:type?', // `/:type?` can be omitted since here we don't match it
matcher: {
from: [
{ action: 'qs', key: 'fs' },
{ action: 'regex', match: '^([^,]+)' },
],
to: [
{ action: 'qs', key: 'ts' },
{ action: 'regex', match: '^([^,]+)' },
],
date: [
{ action: 'qs', key: 'date' }
],
},
}
],
},
}A use case of remap and overrideTitle
Old rule
{
'423down.com': {
_name: '423down',
www: [
{
title: '安卓软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'apk') {
return '/423down/android/apk';
}
}
},
{
title: '原创软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'zd423') {
return '/423down/computer/originalsoft';
}
}
},
{
title: '媒体播放',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'multimedia') {
return '/423down/computer/multimedia';
}
}
},
{
title: '网页浏览',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'browser') {
return '/423down/computer/browser';
}
}
},
{
title: '图形图像',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'image') {
return '/423down/computer/image';
}
}
},
{
title: '聊天软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'im') {
return '/423down/computer/im';
}
}
},
{
title: '办公软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'work') {
return '/423down/computer/work';
}
}
},
{
title: '上传下载',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'down') {
return '/423down/computer/down';
}
}
},
{
title: '系统辅助',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'systemsoft') {
return '/423down/computer/systemsoft';
}
}
},
{
title: '系统必备',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'systemplus') {
return '/423down/computer/systemplus';
}
}
},
{
title: '安全软件',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'security') {
return '/423down/computer/security';
}
}
},
{
title: '补丁相关',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'patch') {
return '/423down/computer/patch';
}
}
},
{
title: '硬件相关',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'hardwork') {
return '/423down/computer/hardware';
}
}
},
{
title: 'windows 11',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'win11') {
return '/423down/os/win11';
}
}
},
{
title: 'windows 10',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'win10') {
return '/423down/os/win10';
}
}
},
{
title: 'windows 7',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'win7') {
return '/423down/os/win7';
}
}
},
{
title: 'windows xp',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'winxp') {
return '/423down/os/winxp';
}
}
},
{
title: 'windows pe',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
target: (params) => {
if (params.type === 'winpe') {
return '/423down/os/winpe';
}
}
}
]
},
}New rule
{
'423down.com': {
_name: '423down',
www: [
{
title: '分类',
docs: 'https://docs.rsshub.app/bbs.html#_423down',
source: '/:type',
route: '/423down/:category_and_type', // original route is `/423down/:category/:type`
matcher: {
category_and_type: [
{ action: 'path', key: 'type', stopIfEmpty: false },
{
action: 'remap',
map: {
'': { value: 'index/all', overrideTitle: '首页' },
apk: { value: 'android/apk', overrideTitle: '安卓软件' },
zd423: { value: 'computer/originalsoft', overrideTitle: '原创软件' },
multimedia: { value: 'computer/multimedia', overrideTitle: '媒体播放' },
browser: { value: 'computer/browser', overrideTitle: '网页浏览' },
image: { value: 'computer/image', overrideTitle: '图形图像' },
im: { value: 'computer/im', overrideTitle: '聊天软件' },
work: { value: 'computer/work', overrideTitle: '办公软件' },
down: { value: 'computer/down', overrideTitle: '上传下载' },
systemsoft: { value: 'computer/systemsoft', overrideTitle: '系统辅助' },
systemplus: { value: 'computer/systemplus', overrideTitle: '系统必备' },
security: { value: 'computer/security', overrideTitle: '安全软件' },
patch: { value: 'computer/patch', overrideTitle: '补丁相关' },
hardwork: { value: 'computer/hardware', overrideTitle: '硬件相关' },
win11: { value: 'os/win11', overrideTitle: 'windows 11' },
win10: { value: 'os/win10', overrideTitle: 'windows 10' },
win7: { value: 'os/win7', overrideTitle: 'windows 7' },
winxp: { value: 'os/winxp', overrideTitle: 'windows xp' },
winpe: { value: 'os/winpe', overrideTitle: 'windows pe' },
}
}
]
}
}
]
},
}More Information
Here is a filtered radar-rules.js containing only those rules using target function (based on https://github.com/DIYgod/RSSHub/blob/780031a5e3e6a43691bb877e759a152cc4c8779f/build/radar-rules.js): radar-rules-filtered.tar.gz (.js ext name is banned by GitHub, so I compressed it)
You can filter it by yourself, here's some vim magic:
:%s/\v\{\_[^{}()]+title:\_[^{}()]+},?//g
:%s/\v['"][^"']+['"]: ?\{\_[^{}]+},?//g
:%s/\v\w+: ?\[\_\s*],?//g
:%g/^\s*$/dcc @NeverBehave @Cay-Zhang @LeetaoGoooo @TonyRL
Source: DIYgod/RSSHub-Radar