[Bug]: Reindexing action has poor flow for archival processes.
I've read the documentation
- I'm running the latest version of Tube Archivist and have read the release notes.
- I have read the how to open an issue guide, particularly the bug report section.
Operating System
Linux and Synology
Your Bug Report
Describe the bug
With the increase in YT actions to throttle responses via various means, including providing partial or wrong responses, the reindex action's priority of deleting metadata, subtitles, and/or thumbnails/art first, then downloading a latest copy is harmful for archival purposes and leaves a lot of channels and videos without previously available and downloaded artifacts.
Steps To Reproduce
Perform reindex action.
Expected behavior
Reindex should download first, validate the results (if possible), then replace the existing artifacts with the updated artifacts.
Relevant log output
tubearchivist | [2025-03-28 08:18:32,033: WARNING/ForkPoolWorker-2] -sFc6pscbYk: get metadata from es
tubearchivist | [2025-03-28 08:18:32,047: WARNING/ForkPoolWorker-2] -sFc6pscbYk: get metadata from youtube
tubearchivist | [2025-03-28 08:18:42,666: WARNING/ForkPoolWorker-2] UCH6vXjt-BA7QHl0KnfL-7RQ: get metadata from es
tubearchivist | [2025-03-28 08:18:42,973: WARNING/ForkPoolWorker-2] -sFc6pscbYk: delete subtitles
tubearchivist | [2025-03-28 08:18:43,299: WARNING/ForkPoolWorker-2] -sFc6pscbYk-en: get user uploaded subtitles
tubearchivist | [2025-03-28 08:18:43,299: WARNING/ForkPoolWorker-2] -sFc6pscbYk-no: get user uploaded subtitles
tubearchivist | [2025-03-28 08:18:43,300: WARNING/ForkPoolWorker-2] -sFc6pscbYk-no: get auto generated subtitles
tubearchivist | [2025-03-28 08:18:46,367: WARNING/ForkPoolWorker-2] -sFc6pscbYk: get comments
tubearchivist | [2025-03-28 08:19:05,038: WARNING/ForkPoolWorker-2] WARNING: [youtube] Incomplete data received. Retrying (1/3)...
tubearchivist | [2025-03-28 08:19:05,329: WARNING/ForkPoolWorker-2] WARNING: [youtube] Incomplete data received. Retrying (2/3)...
tubearchivist | [2025-03-28 08:19:05,596: WARNING/ForkPoolWorker-2] WARNING: [youtube] Incomplete data received. Retrying (3/3)...
tubearchivist | [2025-03-28 08:19:05,826: WARNING/ForkPoolWorker-2] WARNING: [youtube] Incomplete data received. Giving up after 3 retries
tubearchivist | [2025-03-28 08:19:14,837: WARNING/ForkPoolWorker-2] -sFc6pscbYk: delete comments
tubearchivist | [2025-03-28 08:19:14,982: WARNING/ForkPoolWorker-2] -sFc6pscbYk: upload comments
tubearchivist | [2025-03-28 08:20:15,232: WARNING/ForkPoolWorker-2] -urpGp15-7Y: get metadata from es
tubearchivist | [2025-03-28 08:20:15,248: WARNING/ForkPoolWorker-2] -urpGp15-7Y: get metadata from youtube
tubearchivist | [2025-03-28 08:20:25,456: WARNING/ForkPoolWorker-2] UCXyVz9-w9Ippr-j2Yz4zAcQ: get metadata from es
tubearchivist | [2025-03-28 08:20:26,506: WARNING/ForkPoolWorker-2] -urpGp15-7Y: delete subtitles
tubearchivist | [2025-03-28 08:20:28,999: WARNING/ForkPoolWorker-2] -urpGp15-7Y-en: get user uploaded subtitles
tubearchivist | [2025-03-28 08:20:28,999: WARNING/ForkPoolWorker-2] -urpGp15-7Y-en: get auto generated subtitles
tubearchivist | [2025-03-28 08:20:28,999: WARNING/ForkPoolWorker-2] -urpGp15-7Y-no: get user uploaded subtitles
tubearchivist | [2025-03-28 08:20:28,999: WARNING/ForkPoolWorker-2] -urpGp15-7Y-no: get auto generated subtitles
tubearchivist | [2025-03-28 08:20:35,266: WARNING/ForkPoolWorker-2] -urpGp15-7Y: get commentsAnything else?
I'll start working on this as it is causing a lot of issues by deleting known-good artifacts and replacing them with either the default (because the download failed) or just deleting them and not having a replacement.
Source: tubearchivist/tubearchivist