flyscrape

Flyscrape is a command-line web scraping tool designed for those without advanced programming skills, enabling precise extraction of website data.

Installation · Documentation · Releases

Demo

Features

• Standalone: Flyscrape comes as a single binary executable.
• jQuery-like: Extract data from HTML pages with a familiar API.
• Scriptable: Use JavaScript to write your data extraction logic.
• System Cookies: Give Flyscrape access to your browsers cookie store.
• Browser Mode: Render JavaScript heavy pages using a headless Browser.
• Nested Scraping: Extract data from linked pages within a single scrape.

Overview

• Example
• Installation
  • Recommended
  • Homebrew
  • Pre-compiled binary
  • Compile from source
• Usage
• Configuration
• Query API
• Flyscrape API
  • Document Parsing
  • File Downloads
• Issues and suggestions

Example

This example scrapes the first few pages form Hacker News, specifically the New, Show and Ask sections.

…

$ flyscrape run hackernews.js
[
  {
    "url": "https://news.ycombinator.com/new",
    "data": {
      "title": "New Links | Hacker News",
      "posts": [
        {
          "title": "Show HN: flyscrape - An standalone and scriptable web scraper",
          "url": "https://flyscrape.com/"
        },
        ...
      ]
    }
  }
]

Check out the examples folder for more detailed examples.

Installation

Recommended

The easiest way to install flyscrape is via its install script.

curl -fsSL https://flyscrape.com/install | bash

Homebrew

For macOS users flyscrape is also available via homebrew:

brew install flyscrape

Pre-compiled binary

flyscrape is available for MacOS, Linux and Windows as a downloadable binary from the releases page.

Compile from source

To compile flyscrape from source, follow these steps:

1. Install Go: Make sure you have Go installed on your system. If not, you can download it from https://go.dev/.

2. Install flyscrape: Open a terminal and run the following command:

   go install github.com/philippta/flyscrape/cmd/flyscrape@latest
   

Usage

Usage:

    flyscrape run SCRIPT [config flags]

Examples:

    # Run the script.
    $ flyscrape run example.js

    # Set the URL as argument.
    $ flyscrape run example.js --url "http://other.com"

    # Enable proxy support.
    $ flyscrape run example.js --proxies "http://someproxy:8043"

    # Follow paginated links.
    $ flyscrape run example.js --depth 5 --follow ".next-button > a"

    # Set the output format to ndjson.
    $ flyscrape run example.js --output.format ndjson

    # Write the output to a file.
    $ flyscrape run example.js --output.file results.json

Configuration

Below is an example scraping script that showcases the capabilities of flyscrape. For a full documentation of all configuration options, visit the documentation page.

…

Query API

…

Flyscrape API

Document Parsing

import { parse } from "flyscrape";

const doc = parse(`
`);
const text = doc.find(".foo").text();

File Downloads

…

Issues and Suggestions

If you encounter any issues or have suggestions for improvement, please submit an issue.