lightnovel-crawler

---

pip install -U lightnovel-crawler

lncrawl crawl "https://example.com/novel/page" -f epub --all

That is the whole download path: it discovers the chapter list, fetches every chapter, and writes an EPUB you can put on an e-reader. The same package also ships a server — one command turns it into a multi-user library with a browser reader, translations, and scheduled re-downloads of the stories you follow.

Currently 361 sources across 11 languages, served by 446 crawlers.

Personal use only. Sites publish fiction under their own terms and copyright. Use this for personal backups of content you have legitimate access to. Do not redistribute or sell someone else's work.

What you get

Hundreds of sites, one interface Every source is a small crawler file with the same contract, so a URL is all you supply. Search runs across every source that supports it. The full list is regenerated by CI. A library, not a downloads folder Shelve novels into libraries, favourite them, and re-download when a story updates. Reading history and progress follow you between devices. Reads in the browser A paginated reader with fonts, themes and text-to-speech — so a download is readable the moment it finishes, without exporting anything first. Translates while it downloads Multi-engine translation with failover, driven by lncrawl-translator. A per-novel glossary keeps character and place names consistent across every chapter. Eighteen output formats EPUB, TXT and JSON on their own; the rest through Calibre. One download can produce several at once, per volume or whole-novel. Made for more than one person Accounts with tiers and quotas, e-mail notification on job completion, announcements, feedback, and admin screens for all of it. Knows why a site said no Blocks are diagnosed rather than retried — see below. A challenge page served as 200 is a failure, not a chapter. Fix a source without leaving the browser The source editor gives you the crawler file with completions and diagnostics from a real language server, and a test run that streams its results back. Scriptable end to end Everything the web app does goes through a documented REST API — OpenAPI at /docs, ReDoc at /redoc.

Install

Pick one. All three ship the same engine, the same sources and the same web app.

Standalone — no Python needed

Platform Download Windows lncrawl.exe Linux lncrawl macOS lncrawl

Run it and the desktop app opens with no login. Older versions are on the releases page.

pip — the CLI and the server

Python 3.9 or newer.

pip install -U lightnovel-crawler
lncrawl -h

If pip fails, try python -m pip install -U lightnovel-crawler. If lncrawl is not on your PATH, python -m lncrawl works the same.

Install straight from the repository

# latest stable
pip install -U git+https://github.com/lncrawl/lightnovel-crawler.git#egg=lightnovel-crawler

# development branch — newest fixes, no stability promise
pip install -U https://github.com/lncrawl/lightnovel-crawler/tarball/refs/heads/dev#egg=lightnovel-crawler

…

Docker — the home server

mkdir -p lncrawl-data
docker run -d --name lncrawl-server \
  -v ./lncrawl-data:/data \
  -p 8181:8181 \
  -e TZ=Europe/London \
  ghcr.io/lncrawl/lightnovel-crawler -ll server

Open http://localhost:8181 and sign in with admin / admin. Change that password in Settings → Account before anyone else can reach the port.

Set TZ. A container's clock is UTC, and a browser whose timezone disagrees with where its address geolocates reads as a robot. Measured in this image: 1 of 3 challenged sites cleared on the default clock, 3 of 3 with TZ set to the zone your address looks like it is in. It matters more than it sounds — see When a site says no.

The web app

lncrawl app opens it as a desktop window with no login. lncrawl server runs it for the network, with accounts — the same thing the Docker image starts.

1. Crawlers lists every supported site, or paste a novel URL straight into the search bar.
2. Pick the chapters — a range, or all of them.
3. Pick the output formats, and a language if you want it translated.
4. Requests tracks the job while it runs.
5. Read it in the browser, or download the file.

Saving a novel to a Library is what makes the next update cheap: re-download fetches only the chapters that appeared since.

The command line

# first 10 chapters as EPUB
lncrawl crawl "https://example.com/novel/page" -f epub --first 10

# everything, in two formats
lncrawl crawl "https://example.com/novel/page" -f epub -f txt --all

# only the chapters that are not downloaded yet
lncrawl crawl "https://example.com/novel/page" -f epub --resume

# find it by title instead
lncrawl search "The Beginning After The End"

# what does this build know about a site?
lncrawl dev explain https://example.com/

# open the app, or serve it
lncrawl app
lncrawl server --port 8181

Any URL from a supported source works. lncrawl <command> -h documents the rest, and lncrawl config reads and writes the settings the web app shows under Settings.

Output formats

Format Calibre Best for epub Most e-readers and reading apps txt Any text editor ️ json Scripts and pipelines pdf ✓ Print-ready, universal azw3 ✓ Kindle (current) mobi ✓ Kindle (older devices) kepub ✓ Kobo docx ✓ Word, LibreOffice rtf ✓ WordPad and others fb2 ✓ FB2 readers htmlz ✓ A book as one HTML bundle lit ✓ MS Reader (obsolete) lrf ✓ Sony readers ️ pdb ✓ PalmOS (legacy) pmlz ✓ eReader / Palm Markup rb ✓ RocketBook / REB1100 tcr ✓ Psion readers snb ✓ Shanda Bambook

The ticked formats are converted from the EPUB by Calibre, which you install separately. On macOS and Linux its ebook-convert is found automatically; on Windows, add the Calibre folder to your Path. A server that would rather not carry Calibre can point the calibre config section at a remote ebook-convert-api instead.

When a site says no

Most of these sites sit behind bot detection, and a downloader that answers a block by retrying harder is a downloader that quietly stops working. This one reads what is blocking before it reacts.

• It escalates only as far as it has to. A plain request first, a different address when the block is about where you came from, a real browser only when a challenge genuinely needs one. Rotating the proxy at a rate limit is the reflex that makes things worse — the new address gets throttled just the same, because the pacing was the problem.
• Challenges are solved in a browser you already have. Firefox for preference, Chrome as the fallback, hidden by default. The clearance is reused, so one solve covers a whole book instead of one page.
• An empty page is a failure, not a success. A challenge screen and a page of adverts both answer 200. Parsed as a chapter they produce a download that reports success and contains nothing, which is the most common way a source breaks.

Ask what it knows about any site:

$ lncrawl dev explain https://aquareader.org/
aquareader.org
  binding layer : L9 Managed JavaScript challenge — reads a hybrid property, solve
  tier          : direct (unproven)