PixivUtil2

Requirements:

• Running from source code:

  • Python 3.10+ (https://www.python.org/)
    • Archive mode required Python 3.13+
  • Additional library listed in requirements.txt
  • IDE Environment: see https://github.com/Nandaka/PixivUtil2/wiki/IDE-Enviroment-(Windows)

• Dependent software

  • FFmpeg (https://www.ffmpeg.org/) - used for converting ugoira to video.
  • VC++ Redistributable - Needed for pyexiv2 to write XMP metadata in Windows (if enabled).

Capabilities:

• Download by member_id
• Download by image_id
• Download by tags
• Download from list (list.txt)
• Download from bookmarked artists (/bookmark.php?type=user) including private/hidden bookmarks.
• Download from bookmarked images (/bookmark.php) including private/hidden bookmarks.
• Download from tags list (tags.txt)
• Download new illustrations from bookmarked artist (/bookmark_new_illust.php)
• Download by Title/Caption
• Download by Tag and Member Id
• Download Member Bookmark (/bookmark.php?id=)
• Download by Group Id
• Download from supported artists (FANBOX)
• Download by artist/creator id (FANBOX)
• Download by post id (FANBOX)
• Download from followed artists (FANBOX)
• Download latest posts from supported artists (FANBOX)
• Re-encoding of all ugoira present in folder
• Batch Download from batch_job.json (experimental) See https://github.com/Nandaka/PixivUtil2/wiki/Using-Batch-Job-(Experimental)
• Manage database:
  • Show all member
  • Show all downloaded images
  • Export list (member_id only)
  • Export list (detailed)
  • Export local database (image_id)
  • Show member by last downloaded date
  • Show image by image_id
  • Show member by member_id
  • Show image by member_id
  • Delete member by member_id
  • Delete image by image_id
  • Delete member and image (cascade deletion)
  • Blacklist image by image_id
  • Show all deleted member
  • Export FANBOX post list
  • Delete FANBOX download history by member_id
  • Delete FANBOX download history by post_id
  • Delete Sketch download history by member_id
  • Delete Sketch download history by post_id
  • Clean Up Database (remove db entry if downloaded file is missing)
• Export user bookmark (member_id) to a text files.

Docker

$ docker build -t pixivutil2 .
$ docker run -it --rm \
  -v $(pwd):/workdir \
  -w /workdir \
  pixivutil2 \
  /bin/bash -c "python PixivUtil2.py"

WARNING

Overusage can lead to Pixiv blocking your IP for a few hours.

FAQs

A. Usage

…

B.Bugs/Source Code/Supports

…

C.Log Messages

…

Command Line Option

Please refer run with --help for latest information.

…

Error Codes

• 100 = Not Logged in.
• 1001 = User ID not exist/deleted.
• 1002 = User Account is Suspended.
• 1003 = Unknown Member Error.
• 1004 = No image found.
• 1005 = Cannot login.
• 2001 = Unknown Error in Image Page.
• 2002 = Not in MyPick List, Need Permission.
• 2003 = Public works can not be viewed by the appropriate level.
• 2004 = Image not found/already deleted.
• 2005 = Image is disabled for under 18, check your setting page (R-18/R-18G).
• 2006 = Unknown Image Error.
• 9000 = Download Failed.
• 9001 = Download Failed: Harddisk related.
• 9002 = Download Failed: Network related.
• 9005 = Server Error.

config.ini

[Authentication]

• username

  Your pixiv username. Needed for OAuth. Please make sure the combination of username and password is valid in case of OAuth error. If you get error 103, please try changing username from pixiv ID to email address or the other way around.

• password

  Your pixiv password, in clear text! Needed for OAuth. Please make sure the combination of username and password is valid in case of OAuth error.

• cookie

  Your cookies for pixiv login, will be automatically updated in the login. See https://github.com/Nandaka/PixivUtil2/issues/814#issuecomment-711182644 for details.

• cookieFanbox

  Cookie for fanbox.cc, normally no need to fill in.

• cookieFanboxTemp

  Workaround for Fanbox's TLS fingerprinting blacklist. See Q.12 of Usage for instructions.

• refresh_token

  Used for OAuth refresh token to avoid relogin too many time. Automatically generated upon succesful OAuth login.

• userAgentImpersonation

  Parametrised argument to customise which browser PixivUtil2 will impersonate when downloading from Fanbox. Defaults to firefox135.

[Pixiv]

• numberofpage

  Number of page to be processed, put 0 to process all pages.

• r18mode

  Only list images tagged R18, for member, member's bookmark, and search by tag. Set to True to enable.

• r18Type

  Allow filtering for R-18 type (R-18 or R-18G) Set r18Type with value 0 = both R18 and R-18G, 1 = only R18, or 2 = only R18G

• dateformat

  Pixiv DateTime format, leave blank to use default format (YYYY-MM-DD). Refer to http://strftime.org/ for syntax. Quick Reference:

  • %d = Day, %m = Month, %Y = Year (4 digit)
  • %H = Hour (24h), %M = Minute, %S = Seconds

• autoAddMember

  Automatically save member id to db for all download.

• autoAddTag

  Automatically add image tags for db for all downloads.

• autoAddCaption

  Automatically save captions for db for all downloads.

• autoAddStats

  Automatically save an artwork's engagement stats (view, like, bookmark, comment and response counts).

• aiDisplayFewer

  if true, filter out AI-generated images from downloading.

[FANBOX]

• filenameFormatFanboxContent

  Similar to filename format, but for files inside FANBOX posts.

• filenameFormatFanboxCover

  Similar to filename format, but for FANBOX post cover images

• filenameFormatFanboxInfo

  Similar to filename format, but for info dumps.

• writeHtml

  A switch to decide whether to write FANBOX posts into HTMLs or not.

  • If set to True, article type posts will for sure be written into HTMLs, while non-article type posts are controlled with minTextLengthForNonArticle and minImageCountForNonArticle.
  • If set to False, no post will be written into HTMLs.
  • filenameFormatFanboxInfo will be used for filename.
  • For HTML format, please refer to 'HTML Format' section

• minTextLengthForNonArticle

  Works with minImageCountForNonArticle. When 'writeHtml' is True, a non-article post should contain text longer than this value to be written into HTML.

• minImageCountForNonArticle

  Works with minTextLengthForNonArticle. When writeHtml is True, a non-article post should contain at least this many files/images to be written into HTML.

• useAbsolutePathsInHtml

  Set to True to use absolute paths in HTMLs. Set to False to use relative paths.

• downloadCoverWhenRestricted

  Set to True to download FANBOX post cover images even if they are restricted.

• checkDBProcessHistory

  Each FANBOX post has a updated_date value, which will be recorded/updated in database after it is processed.

  • When this is True, the values in database would be checked when processing each post. If record is no earlier than the newly retrieved date, which means that the post has not been processed at all or changed since last time, the post would be skipped.
  • Can be combined with checkUpdatedLimitFanbox to stop checking more posts for the current FANBOX member after enough unchanged/already-processed posts are encountered.
  • When this is False, posts will be processed anyways.

• checkUpdatedLimitFanbox

  Skip to next FANBOX member if a number of consecutive previously processed/unchanged posts are encountered for the current member while checkDBProcessHistory is enabled. Set to 0 to disable.

• listPathFanbox

  The list file for fanbox creators. One creator per line. Doesn't support custom path.

[Network]

• useproxy

  Set True to use proxy server, or False to disable it.

• proxyaddress

  Proxy server address, use this format:

  • http://<username>:<password>@<proxy_server>:<port> or
  • socks5://<username>:<password>@<proxy_server>:<port> or
  • socks4://<username>:<password>@<proxy_server>:<port>

• useragent

  Browser user agent to spoof. You can check it from https://www.whatismybrowser.com/detect/what-is-my-user-agent

• userobots

  Download robots.txt for mechanize.

• timeout

  Time to wait before giving up the connection, in seconds.

• retry

  Number of retries.

• retrywait

  Waiting time for each retry, in seconds.

• downloadDelay

  Set random delay up to n seconds for each image post. Set to 0 to disable.

• checkNewVersion

  Set to True to check new releases in github.

• notifyBetaVersion

  Set to False to ignore beta releases.

• openNewVersion

  Set to False to disable opening new releases in browser.

• enableSSLVerification

  Enable SSL verication, only set to False if you always encounter SSL Error (this disable the security)

[Debug]

• logLevel

  Set log level, valid values are CRITICAL, ERROR, WARNING, INFO, DEBUG, and NOTSET

  Two log files are written next to the application: pixivutil.log with everything at the configured level, and pixivutil_error.log with only warnings and errors, so a failed run can be reviewed without searching the full log. Both rotate at 10MB, keeping 10 backups, and both are suppressed by disableLog.

• enableDump

  Enable HTML Dump. Set to False to disable.

• skipDumpFilter

  Skip HTML Dump based on error code (using regex format). E.g.: 1.|2. => skip all HTML dump for error code 1xxx/2xxx.

• dumpMediumPage

  Dump all medium page for debugging. Set to True to enable.

• dumpTagSearchPage

  Dump tags search page for debugging.

• debughttp

  Print http header, useful for debuggin. Set 'False' to disable.

[IrfanView]

• IrfanViewPath

  Set directory where IrfanView is installed (needed to start IrfanView)

• startIrfanView

  Set to True to start IrfanView with downloaded images when exiting pixivUtil

  • This will create download-lists
  • Be sure to set IrfanView to load Unicode-Plugin on startup when there are unicode-named files!

• startIrfanSlide

  Set to True to start IrfanView-Slideshow with downloaded images when exiting pixivUtil.

  • This will create download-lists
  • Be sure to set IrfanView to load Unicode-Plugin on startup when there are unicode-named files!
  • Slideshow-options will be same as you have set in IrfanView before!

• createDownloadLists

  Set to True to automatically create download-lists.

[Settings]

• downloadlistdirectory

  list.txt path, also used for download-lists needed for createDownloadLists and IrfanView-Handling If leaved blank it will create download-lists in pixivUtil-directory.

• uselist

  Set to True to parse list.txt. This will update the DB content from the list.txt (member_id and custom folder).

• processfromdb

  Set True to use the member_id from the DB.

• rootdirectory

  Your root directory for saving the images.

• downloadavatar

  Set to True to download the member avatar as 'folder.jpg'

• usesuppresstags

  Remove the suppressed tags from %tags% meta for filename. The list is taken from suppress_tags.txt, each tags is separated by new line.

• tagsLimit

  Number of tags to be used for %tags% meta in filename. Use -1 to use all tags.

• writeImageJSON

  Set to True to export the compact image information to JSON file. The filename is following filename(Manga)Infoformat + .json. If you want the original info from source, use with writeRawJSON.

• writeimageinfo

  Set to True to export the compact image information to text file. The filename is following filename(Manga)Infoformat + .txt. If you want the original info from source, use with writeRawJSON.

• writeRawJSON

  Set to True to export the original JSON untouched of the image for writeImageJSON.

• RawJSONFilter

  Enter the JSON keys which you want to filter out for writeRawJSON. Keys are seperated by a comma.

• includeSeriesJSON

  Set to True to export the series information