Startpage engine issues (to-do list)

update from the maintainers: we have tagged startpage issue, see Label "startpage"

To-do list for the Startpage engine

Just a list of things to do to potentially fix the issues the Startpage engine has at the moment. I plan on addressing them at some point, but if you do want to work on them comment here so the work doesn't get done twice.

Form data

• Order the query parameters for first page and subsequent pages like a real request.
• Use a new sc code for every search request (currently cached for an hour for all searches)
• lui and language form parameter need to include the full language name (e.g. english instead of en)
• Add support for "all" region (not included at the moment)

Cookies

• params['cookies']['preferences']
  • language, language_ui and search_results_region should always be present
• Add missing locationPref cookie

Response parsing

• Add missing CAPTCHA handling in main response function (only handled in get_sc_code)

Extra notes

• abd and abe stand for "adblock detected" and "adblock enabled", they are both set to 0 if javascript is disabled (or there's no adblocker).
• Startpage has the possibility of making first page searches without an sc code (via GET requests, opensearch config and extension have this)
• Depending on if we want to mimic a search query that uses javascript or without, there are some obvious pings on response here:
  • No javascript: a request is sent to https://www.startpage.com/sp/nojst?bdst=[token]
  • Javascript: a request is sent to https://www.startpage.com/sp/footerImage/cdn/images/startpage-logo-gradient.svg?sc=[code]&cdw=1&rpw=1 (&cdw=1&rpw=1 are included on adblock detection)