Adopt WebDriverAbstract as a solution for active (JavaScript) websites

Hello everyone,

A few weeks ago I came across the well-known problem that rss-bridge doesn't work for some websites. This is always the case when the website loads some content via XMLHttpRequest (XHR) and / or changes it using JavaScript. After some experimentation and thinking about the problem, I came to the conclusion that it can only be solved using a web browser. Coincidentally, I already had a lot of experience with Selenium and the WebDriver and decided to come up with a solution.

I now suggest that the WebDriverAbstract class be imported into the project, temporarily in its own branch. As examples I have written two bridges that use these.

The following problems still need to be solved:

• I didn't commit my composer.lock because the changes look random and I didn't want to invest time (yet) to understand composer.
• I have added the php-webdriver dependencies to bootstrap.php, but they are not complete. They look messy and unsorted because of the dependencies between them. I suggest to use the autoloader.
• All bridges that depend on WebDriverAbstract would create high(er) load on a multi-user (open) server. Maybe there should be an option to disable the class instead of each bridge using it.
• Because loading web pages through a browser is such a heavier load, it would make sense to stop scraping as soon as an element is in a cache (assuming the elements are sorted chronologically). Is there already a cache for elements that have already been scraped? How can I use it? I looked around the code but didn't find any examples of this or documentation.
• What do you miss in the documentation for WebDriverAbstract?

Kind regards,

Holger