Feature Request: [Pipeline Resilience] Fallback to dynamic/future Wayback Machine URL when archive_org extractor fails
What type of suggestion are you making?
New extractor / type of content to save
What is the problem that your feature request solves?
When using ArchiveBox in automated data pipelines, we rely on the SUBMIT_ARCHIVE_DOT_ORG=True feature to generate a permanent public snapshot URL. However, the Internet Archive "Save Page Now" API for some cases in batch procesing ,frequently returns 520, 502, or 429 errors due to their server load. When this happens, the extractor fails, and our database pipelines are left without a public Wayback URL, forcing us to build retry logic.
What is your proposed solution?
I propose adding a fallback mechanism (perhaps behind a config flag like USE_WAYBACK_FALLBACK_URL=True).
What hacks or alternative solutions have you tried to solve the problem?
Currently, I am wrapping the archivebox add CLI command in a Python script. My script parses the stdout logs, detects the 520 error, and manually generates fallback URL to save into the database. It would be much cleaner if the ArchiveBox handled this natively.
Share the entire output of the archivebox version command for the current version you are using.
PS C:\Users\HP\Desktop\Archive box> docker compose run archivebox version
time="2026-05-06T15:07:23+01:00" level=warning msg="C:\\Users\\HP\\Desktop\\Archive box\\docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion"
0.7.3
ArchiveBox v0.7.3 COMMIT_HASH=069aabc BUILD_TIME=2024-12-15 09:54:03 1734256443
IN_DOCKER=True IN_QEMU=False ARCH=x86_64 OS=Linux PLATFORM=Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.36 PYTHON=Cpython
FS_ATOMIC=True FS_REMOTE=True FS_USER=911:911 FS_PERMS=644
DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND=ripgrep LDAP=False
[i] Dependency versions:
√ PYTHON_BINARY v3.11.11 valid /usr/local/bin/python3.11
√ SQLITE_BINARY v2.6.0 valid /usr/local/lib/python3.11/sqlite3/dbapi2.py
√ DJANGO_BINARY v3.1.14 valid /usr/local/lib/python3.11/site-packages/django/__init__.py
√ ARCHIVEBOX_BINARY v0.7.3 valid /usr/local/bin/archivebox
√ CURL_BINARY v8.10.1 valid /usr/bin/curl
√ WGET_BINARY v1.21.3 valid /usr/bin/wget
√ NODE_BINARY v20.18.1 valid /usr/bin/node
√ SINGLEFILE_BINARY v1.1.54 valid /app/node_modules/single-file-cli/single-file
- READABILITY_BINARY - disabled /app/node_modules/readability-extractor/readability-extractor
- MERCURY_BINARY - disabled /app/node_modules/@postlight/parser/cli.js
- GIT_BINARY - disabled /usr/bin/git
- YOUTUBEDL_BINARY - disabled /usr/local/bin/yt-dlp
√ CHROME_BINARY v131.0.6778.33 valid /usr/bin/chromium-browser
√ RIPGREP_BINARY v13.0.0 valid /usr/bin/rg
[i] Source-code locations:
√ PACKAGE_DIR 23 files valid /app/archivebox
√ TEMPLATES_DIR 3 files valid /app/archivebox/templates
- CUSTOM_TEMPLATES_DIR - disabled None
[i] Secrets locations:
- CHROME_USER_DATA_DIR - disabled None
- COOKIES_FILE - disabled None
[i] Data locations:
√ OUTPUT_DIR 5 files @ valid /data
√ SOURCES_DIR 317 files valid ./sources
√ LOGS_DIR 1 files valid ./logs
√ ARCHIVE_DIR 132 files valid ./archive
√ CONFIG_FILE 81.0 Bytes valid ./ArchiveBox.conf
√ SQL_INDEX 1.8 MB valid ./index.sqlite3How badly do you want this new feature?
- It's an urgent deal-breaker, I can't live without it
- It's important to add it in the near-mid term future
- It would be nice to have eventually
- I'm willing to start a PR to develop this myself
- I have donated money to go towards fixing this issue
Mini Survey
- I like ArchiveBox so far / would recommend it to a friend
- I've had a lot of difficulty getting ArchiveBox set up
- I would pay $10/mo for a hosted version of ArchiveBox if it had this feature
Source: ArchiveBox/ArchiveBox