Update Pi-hole's lists from remote sources easily
Update Pi-hole's lists from remote sources easily
When using remote lists like this it's a hassle to manually check for changes and update - this script will do that for you!
User-created entries will not be touched and those removed from the remote list will be disabled instead.
[!WARNING] If you're not using remote lists like the ones mentioned above then this script will be useless to you - Pi-hole already updates the lists weekly automatically.
filter pdo pdo_sqlite curl openssl intl pcntl posix)Sometimes the base PHP package may already include certain extensions because each distribution package it differently.
PHP install commands for known distros/environments:
Debian: apt install php-cli php-sqlite3 php-curl php-intl
Alpine: apk add php php-pdo_sqlite php-curl php-openssl php-intl php-pcntl php-posix
Arch: pacman -S php php-sqlite
Fedora: dnf install php-cli php-pdo php-intl php-process
CentOS: yum install php-cli php-pdo php-intl php-process
Entware: opkg install php8-cli php8-mod-filter php8-mod-pdo-sqlite php8-mod-curl php8-mod-openssl php8-mod-intl php8-mod-pcntl
[!NOTE] Docker users - look below.
This command will install this script to /usr/local/sbin:
wget -O - https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/install.sh | sudo bashAlternatively you can clone this repo and sudo bash install.sh.
If systemd is available this script will also add service and timer unit files to the system, otherwise a crontab entry in /etc/cron.d/pihole-updatelists will be created.
[!NOTE] If for some reason the install script does not copy service and timer files while your distro has systemd scheduler available you can force the installation by passing
systemdas a parameter to the install script - modifying the install command above withsudo bash -s systemdinstead.
Note that in most cases you will be able to execute this script globally as pihole-updatelists command but some will require you to add /usr/local/sbin to $PATH or execute it via /usr/local/sbin/pihole-updatelists.
[!IMPORTANT] This script does nothing by default (except running
pihole updateGravity) - you have to configure it.
[!TIP] To quickly update the script run
sudo pihole-updatelists --updatewhich will check for script differences and re-run the install script when needed.
[!TIP] If you don't plan on updating blocklists or want to keep Pi-hole's gravity update schedule you should skip this section and set
UPDATE_GRAVITY=falsein the configuration file.
You should disable entry with pihole updateGravity command in /etc/cron.d/pihole as this script already runs it:
sudo nano /etc/cron.d/piholePut a # before this line (numbers might be different):
#49 4 * * 7 root PATH="$PATH:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.logAlternatively, the following sed command will disable the same entry:
sudo sed -e '/pihole updateGravity/ s/^#*/#/' -i /etc/cron.d/piholeYou might have to do this manually after each Pi-hole update.
[!TIP] You can override
pihole-FTL.serviceto disable the cron entry automatically after each update:sudo systemctl edit pihole-FTL.service[Service] ExecStartPre=-+/bin/sh -c "[ -w /etc/cron.d/pihole ] && /bin/sed -e '/pihole[[:space:]]updateGravity/ s/^#*/#/' -i /etc/cron.d/pihole"
If you already imported any of the remote lists manually you should migrate their entries to allow the script to disable them in case they are removed from the remote list.
If you used pihole5-list-tool to import adlists and whitelist you can use these commands to do this quickly:
sudo sqlite3 /etc/pihole/gravity.db "UPDATE adlist SET comment = 'Managed by pihole-updatelists' WHERE comment LIKE '%Firebog |%' OR comment LIKE '%[ph5lt]'"
sudo sqlite3 /etc/pihole/gravity.db "UPDATE domainlist SET comment = 'Managed by pihole-updatelists' WHERE comment LIKE '%AndeepND |%' OR comment LIKE '%[ph5lt]'"(code up to date as of pihole5-list-tool 0.6.0)
Alternatively, some manual work is required - pick one:
COMMENT variable in Configuration section) (recommended but might be a lot of work)sudo pihole -g -r recreatesudo pihole-updatelists to finish upUPDATE_GRAVITY=false) run pihole updateGravityFollow the official instructions but use jacklul/pihole:latest image instead, pass configuration variables as environment variables in docker-compose.yml.
If you need to pull a specific version of Pi-hole image you have no other choice but to use custom Dockerfile.
Use jacklul/pihole:latest image instead of pihole/pihole:latest. Version-specific tags are also available but keep in mind they will contain version of the script that was available at the time of that particular version.
If you don't want to use my image you can write custom Dockerfile based on the one in this repository:
FROM pihole/pihole:latest
RUN apk add --no-cache php php-pdo_sqlite php-curl php-openssl php-intl php-pcntl php-posix
RUN wget -O - https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/install.sh | bash -s dockerThen build your image locally and use that image in your docker-composer.yml or launch command line.
You will have to update your local image manually each time update is released.
Your docker-compose.yml file should look similar to this:
…(for more up to date docker-compose.yml see pi-hole/docker-pi-hole)
If you already have existing gravity.db you should also check out Migrating lists and domains section, keep in mind that you will have to adjust paths in the commands mentioned there.
Default configuration file is /etc/pihole-updatelists.conf.
sudo nano /etc/pihole-updatelists.conf| Variable | Default | Description |
|---|---|---|
| BLOCKLISTS_URL | " " | Remote list URL containing list of blocklists to import URLs to single lists should be added manually |
| ALLOWLISTS_URL | " " | Remote list URL containing list of allowlists to import URLs to single lists should be added manually |
| WHITELIST_URL | " " | Remote list URL containing exact domains to whitelist This is specifically for handcrafted lists only, do not use regular allowlists here! |
| REGEX_WHITELIST_URL | " " | Remote list URL containing regex rules for whitelisting |
| BLACKLIST_URL | " " | Remote list URL containing exact domains to blacklist This is specifically for handcrafted lists only, do not use regular blocklists here! |
| REGEX_BLACKLIST_URL | " " | Remote list URL containing regex rules for blacklisting |
| COMMENT | "Managed by pihole-updatelists" | Comment string used to know which entries were created by the script You can still add your own comments to individual entries as long you keep this string intact |
| GROUP_ID | 0 | Assign additional group to all inserted entries, to assign only the specified group (do not add to the default) make the number negative 0 is the default group, you can view ID of the group in Pi-hole's web interface by hovering mouse cursor over group name field on the 'Group management' page Multiple groups are not supported |
| PERSISTENT_GROUP | true | Makes sure entries have the specified group assigned on each script run This does not prevent you from assigning more groups through the web interface but can remove entries from the default group if GROUP_ID is a negative number When disabled but an entry has no groups assigned and is about to be enabled then it will be re-added to the groups it's supposed to be in WARNING: This option might be buggy when running multiple different configurations with same lists |
| REQUIRE_COMMENT | true | Prevent touching entries not created by this script by comparing comment field When false any user-created entry will be disabled, only those created by the script will be active |
| MIGRATION_MODE | 1 | Decides how to migrate disabled entries from another config sections 1 - replace comment field 2 - append to comment field 0 - disables migration, entry will be ignored |
| GROUP_EXCLUSIVE | false | Causes defined group in GROUP_ID to contain one defined list exclusively - only entries from the last list inserted will be enabled This option is experimental |
| UPDATE_GRAVITY | true | Update gravity after lists are updated? (runs pihole updateGravity) When false invokes lists reload instead Set to null to do nothing |
| WARMUP_DNS_CACHE | false | Resolve all blocklist/allowlist domains using local resolver before running gravity update This can fix "list was inaccessible" warnings in Pi-hole when using recursive resolver like Unbound |
| VERBOSE | false | Show more information while the script is running |
| DEBUG | false | Show debug messages for troubleshooting purposes If you're having issues - this might help tracking it down |
| DOWNLOAD_TIMEOUT | 60 | Maximum time in seconds one list download can take before giving up You should increase this when downloads fail because of timeout |
| IGNORE_DOWNLOAD_FAILURE | false | Ignore download failures when using multiple lists This will cause entries from the lists that failed to download to be disabled |
| GRAVITY_DB | "/etc/pihole/gravity.db" | Path to gravity.db in case you need to change it |
| LOCK_FILE | "/var/lock/pihole-updatelists.lock" | Process lockfile to prevent multiple instances of the script from running You shouldn't change it - unless /var/lock is unavailable |
| LOG_FILE | " " | Log console output to file In most cases you don't have to set this as you can view full log in the system journal Put - before path to overwrite file instead of appending to it |
| PIHOLE_CMD | "/usr/local/bin/pihole" | Path to pihole script, Change this only if it isn't in the default location |
| GIT_BRANCH | "master" | Branch to pull remote checksum and update from |
[!NOTE] String values should be put between
" ", otherwise weird things might happen.
[!TIP] You can also give paths to the local files instead of URLs, for example setting
WHITELIST_URLto/home/pi/whitelist.txtwill fetch this file from filesystem.
It is also possible to load configuration variables from the environment by using --env parameter - this will overwrite values in default section of the config file.
[!IMPORTANT] Some variables will have to be prefixed
No open issues yet, or sync has not completed.