[Bug]: ext_pillar with gitfs and gitcli returns empty response.
Author: bebeheiCreated Sep 7, 2026Updated Sep 18, 2026
Labelsbugneeds-triage
What happened?
I tried to upgrade from 3007 to 3008. It does not work, because the pillar data returned by pillar.items is empty.
I changed from pygit2 to gitcli provider explicitly.
3007 configuration:
fileserver_backend:
- git
gitfs_update_interval: 30
gitfs_base: main
gitfs_remotes:
- "ssh://[email protected]/repo-sls.git":
- pubkey: /etc/salt/ssh-creds/id_ed25519.pub
- privkey: /etc/salt/ssh-creds/id_ed25519
ext_pillar:
- git:
- __env__ ssh://[email protected]/repo-pillar.git:
- base: main
- pubkey: /etc/salt/ssh-creds/id_ed25519.pub
- privkey: /etc/salt/ssh-creds/id_ed255193008 configuration:
# explicitly use the `git` CLI commands, instead of pygit, Finally we get rid of pygit2
gitfs_provider: gitcli
git_pillar_provider: gitcli
fileserver_backend:
- gitfs
gitfs_update_interval: 30
gitfs_base: main
git_pillar_base: main
pillarenv_from_saltenv: True
pillar_raise_on_missing: True
git_pillar_branch: main
gitfs_remotes:
- "ssh://[email protected]/repo-sls.git"
ext_pillar:
- git:
- __env__ ssh://[email protected]/repo-pillar.git:
- base: mainsalt -l debug minionXY pillar.items pillarenv=base
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/main.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/main.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: nbg-saltmaster-01
[DEBUG ] Missing configuration file: /root/.saltrc
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] tracing.configure called but tracing.enabled is false (pid=867863, service=)
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/main.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/main.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: nbg-saltmaster-01
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] Using selector: EpollSelector
[DEBUG ] Using selector: EpollSelector
[DEBUG ] Marking 'base64_encode' as a jinja filter
[DEBUG ] Marking 'base64_decode' as a jinja filter
[DEBUG ] Marking 'md5' as a jinja filter
[DEBUG ] Marking 'sha1' as a jinja filter
[DEBUG ] Marking 'sha256' as a jinja filter
[DEBUG ] Marking 'sha512' as a jinja filter
[DEBUG ] Marking 'hmac' as a jinja filter
[DEBUG ] Marking 'hmac_compute' as a jinja filter
[DEBUG ] Marking 'random_hash' as a jinja filter
[DEBUG ] Marking 'rand_str' as a jinja filter
[DEBUG ] Marking 'file_hashsum' as a jinja filter
[DEBUG ] Marking 'http_query' as a jinja filter
[DEBUG ] Marking 'ifelse' as a jinja global
[DEBUG ] Marking 'strftime' as a jinja filter
[DEBUG ] Marking 'date_format' as a jinja filter
[DEBUG ] Marking 'raise' as a jinja global
[DEBUG ] Marking 'match' as a jinja test
[DEBUG ] Marking 'equalto' as a jinja test
[DEBUG ] Marking 'skip' as a jinja filter
[DEBUG ] Marking 'sequence' as a jinja filter
[DEBUG ] Marking 'to_bool' as a jinja filter
[DEBUG ] Marking 'indent' as a jinja filter
[DEBUG ] Marking 'quote' as a jinja filter
[DEBUG ] Marking 'regex_escape' as a jinja filter
[DEBUG ] Marking 'regex_search' as a jinja filter
[DEBUG ] Marking 'regex_match' as a jinja filter
[DEBUG ] Marking 'regex_replace' as a jinja filter
[DEBUG ] Marking 'uuid' as a jinja filter
[DEBUG ] Marking 'unique' as a jinja filter
[DEBUG ] Marking 'min' as a jinja filter
[DEBUG ] Marking 'max' as a jinja filter
[DEBUG ] Marking 'avg' as a jinja filter
[DEBUG ] Marking 'union' as a jinja filter
[DEBUG ] Marking 'intersect' as a jinja filter
[DEBUG ] Marking 'difference' as a jinja filter
[DEBUG ] Marking 'symmetric_difference' as a jinja filter
[DEBUG ] Marking 'method_call' as a jinja filter
[DEBUG ] Marking 'tojson' as a jinja filter
[DEBUG ] Marking 'yaml_dquote' as a jinja filter
[DEBUG ] Marking 'yaml_squote' as a jinja filter
[DEBUG ] Marking 'yaml_encode' as a jinja filter
[DEBUG ] Popen(['git', 'version'], cwd=/home/benedikt.heine, stdin=None, shell=False, universal_newlines=False)
[DEBUG ] Popen(['git', 'version'], cwd=/home/benedikt.heine, stdin=None, shell=False, universal_newlines=False)
[DEBUG ] PubClient connecting to <salt.transport.tcp.PublishClient object at 0x7f8ca6fc30e0> '/var/run/salt/master/master_event_pub.ipc'
[DEBUG ] PubClient connected to <salt.transport.tcp.PublishClient object at 0x7f8ca6fc30e0> '/var/run/salt/master/master_event_pub.ipc'
[DEBUG ] Closing AsyncReqChannel instance
[DEBUG ] The functions from module 'local_cache' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] Reading minion list from /var/cache/salt/master/jobs/40/7dd0703cdcbd187b4c9eb83779fa33a5e5759af9ab4e105f2f4cccd1bd867a/.minions.p
[DEBUG ] get_iter_returns for jid 20260907163442373578 sent to {'minionXY'} will timeout at 16:34:47.393519
[DEBUG ] Skipping non return event: salt/job/20260907163442373578/publish
[DEBUG ] jid 20260907163442373578 return from minionXY
[DEBUG ] return event: {'minionXY': {'ret': {}, 'retcode': 0, 'jid': '20260907163442373578'}}
[DEBUG ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded nested.output
minionXY:
----------
[DEBUG ] jid 20260907163442373578 found all minions {'minionXY'}
[DEBUG ] The functions from module 'localfs_key' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded localfs_key.list_all$> salt-run -l trace git_pillar.update
[TRACE ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'info', 'log_format': '%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'salt'}
[TRACE ] Processing <bound method SaltfileMixIn.process_saltfile of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[TRACE ] Processing <bound method ConfigDirMixIn.process_config_dir of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/main.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/main.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: nbg-saltmaster-01
[DEBUG ] Missing configuration file: /root/.saltrc
[TRACE ] Processing <bound method JIDMixin.process_jid of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_output of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_output_file of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[TRACE ] Processing <bound method OutputOptionsMixIn.process_state_verbose of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[TRACE ] Processing <bound method NoParseMixin.process_no_parse of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f965efa6610>, <CustomOption at 0x7f965a1384b0: --out-indent/--output-indent>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f965e79a8d0>, <CustomOption at 0x7f965a138910: --out-file-append/--output-file-append>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f9660997060>, <CustomOption at 0x7f965a1389f0: --no-color/--no-colour>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f965a1710c0>, <CustomOption at 0x7f965a138ad0: --force-color/--force-colour>)
[TRACE ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f965a171170>, <CustomOption at 0x7f965a138d70: --state-output/--state_output>)
[TRACE ] Processing <bound method LogLevelMixIn.process_log_level of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[TRACE ] Processing <bound method LogLevelMixIn.process_log_file of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[TRACE ] Processing <bound method LogLevelMixIn.process_log_level_logfile of <salt.cli.run.SaltRun object at 0x7f965a9cee40>>
[TRACE ] Processing <function OutputOptionsMixIn._mixin_after_parsed at 0x7f965a13dc70>
[TRACE ] Processing <function SaltRunOptionParser._mixin_after_parsed at 0x7f965a13f7f0>
[TRACE ] Processing <function LogLevelMixIn.__setup_logging_routines at 0x7f965a13c1a0>
[TRACE ] Processing <function MergeConfigMixIn.__merge_config_with_cli at 0x7f965a137a00>
[TRACE ] Processing <function LogLevelMixIn.__setup_console_logger_config at 0x7f965a13c510>
[TRACE ] Processing <function LogLevelMixIn.__setup_logfile_logger_config at 0x7f965a13c5c0>
[TRACE ] Processing <function LogLevelMixIn.__setup_logging_config at 0x7f965a13c670>
[TRACE ] Processing <function LogLevelMixIn.__verify_logging at 0x7f965a13c7d0>
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[TRACE ] Processing <function LogLevelMixIn.__setup_logging at 0x7f965a13c720>
[TRACE ] Setting up console logging: {'log_level': 'trace', 'log_format': '[%(levelname)-8s] %(message)s', 'date_format': '%H:%M:%S'}
[TRACE ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'info', 'log_format': '%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'salt'}
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] Popen(['git', 'version'], cwd=None, stdin=None, shell=False, universal_newlines=False)
[DEBUG ] Popen(['git', 'version'], cwd=None, stdin=None, shell=False, universal_newlines=False)
[DEBUG ] The functions from module 'git_pillar' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded git_pillar.update
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/main.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/main.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: nbg-saltmaster-01
[DEBUG ] Grains refresh requested. Refreshing grains.
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/main.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/main.conf
[TRACE ] 'drill' could not be found in the following search path: '['/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin', '/snap/bin']'
[TRACE ] 'drill' could not be found in the following search path: '['/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin', '/snap/bin']'
[DEBUG ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'package' are being loaded by dir() on the loaded module
[TRACE ] Error loading grains.pending_reboot: 'pending_reboot' grain only available on Windows
[DEBUG ] The functions from module 'resources' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'truststore' are being loaded by dir() on the loaded module
[TRACE ] Loading core.append_domain grain
[TRACE ] Loading core.cwd grain
[TRACE ] Loading core.default_gateway grain
[TRACE ] Loading core.dns grain
[TRACE ] Loading core.fibre_channel_host grain
[TRACE ] Loading core.fqdns grain
[TRACE ] Loading core.get_machine_id grain
[TRACE ] Loading core.get_master grain
[TRACE ] Loading core.get_server_id grain
[TRACE ] Loading core.hostname grain
[TRACE ] Loading core.hwaddr_interfaces grain
[TRACE ] Loading core.id_ grain
[TRACE ] Loading core.ip4_interfaces grain
[TRACE ] Loading core.ip6_interfaces grain
[TRACE ] Loading core.ip_fqdn grain
[TRACE ] Loading core.ip_interfaces grain
[TRACE ] Loading core.kernelparams grain
[TRACE ] Loading core.locale_info grain
[TRACE ] Loading core.os_data grain
[TRACE ] Adding systemd grains
[TRACE ] Adding init grain
[TRACE ] Getting lsb_release distro information
[TRACE ] lsb_release python bindings not available
[TRACE ] Attempting to parse /etc/lsb-release
[TRACE ] Getting OS name, release, and codename from freedesktop_os_release
[TRACE ] Getting OS name, release, and codename from distro id, version, codename
[TRACE ] 'virt-what' could not be found in the following search path: '['/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin', '/snap/bin']'
[TRACE ] Loading core.path grain
[TRACE ] Loading core.pythonexecutable grain
[TRACE ] Loading core.pythonpath grain
[TRACE ] Loading core.pythonversion grain
[TRACE ] Loading core.saltpath grain
[TRACE ] Loading core.saltversion grain
[TRACE ] Loading core.saltversioninfo grain
[TRACE ] Loading core.zmqversion grain
[TRACE ] Loading disks.disks grain
[TRACE ] Device sr0 reports itself as an HDD
[TRACE ] Device sda reports itself as an HDD
[TRACE ] Loading extra.config grain
[TRACE ] Loading extra.get_secure_boot_path grain
[TRACE ] Loading extra.shell grain
[TRACE ] Loading extra.transactional grain
[TRACE ] 'transactional-update' could not be found in the following search path: '['/usr/local/sbin', '/usr/local/bin', '/usr/sbin', '/usr/bin', '/sbin', '/bin', '/snap/bin']'
[TRACE ] Loading extra.uefi grain
[TRACE ] Loading lvm.lvm grain
[TRACE ] Loading mdadm.mdadm grain
[TRACE ] Loading minion_process.grains grain
[TRACE ] Loading opts.opts grain
[TRACE ] Loading package.package grain
[TRACE ] Loading resources.resources grain
[TRACE ] Loading truststore.ca_truststore grain
[DEBUG ] The functions from module 'auth' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'batch' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'cache' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'cluster' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'doc' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'error' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'event' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'fileserver' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'http' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'jobs' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'manage' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'match' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'mine' are being loaded by dir() on the loaded module
[TRACE ] Error loading runners.net: The napalm module could not be imported
[DEBUG ] The functions from module 'network' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'pillar' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'pki' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'queue' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'reactor' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'resource' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'salt' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'sdb' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'ssh' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'survey' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'test' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'winrepo' are being loaded by dir() on the loaded module
[DEBUG ] Using selector: EpollSelector
[DEBUG ] Using selector: EpollSelector
[DEBUG ] Publisher connecting to /var/run/salt/master/master_event_pull.ipc
[DEBUG ] gitcli git_pillar_provider enabled
[DEBUG ] register_cleanup_finalize_function entry, function=<function gitfs_finalize_cleanup at 0x7f9657db59b0>, args=('/var/cache/salt/master',), kwargs={}
[DEBUG ] GitCLI running: git -c core.quotepath=false rev-parse --is-bare-repository in /var/cache/salt/master/git_pillar/6UMN6PLOTjtzIOfVZQddUm_lEiq5dnEwwtfYWjzDVfo=/main
[INFO ] Set update lock for git_pillar remote '__env__ ssh://git@gitlab-internal/repo-pillar.git' on machine_id '6604054a6d6a47a8ae7c1b8c396ca313'
[DEBUG ] Fetching git_pillar remote '__env__ ssh://git@gitlab-internal/repo-pillar.git'
[DEBUG ] GitCLI running: git -c core.quotepath=false fetch --prune --quiet origin --depth 1 +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* in /var/cache/salt/master/git_pillar/6UMN6PLOTjtzIOfVZQddUm_lEiq5dnEwwtfYWjzDVfo=/main
[DEBUG ] Attempting to remove 'update' lock for 'git_pillar' remote '__env__ ssh://git@gitlab-internal/repo-pillar.git' due to lock_set1 'True' or lock_set2 'True'
[DEBUG ] Removed update lock for git_pillar remote '__env__ ssh://git@gitlab-internal/repo-pillar.git' on machine_id '6604054a6d6a47a8ae7c1b8c396ca313'
[DEBUG ] Attempt to remove lock ssh://git@gitlab-internal/repo-pillar.git for file (/var/cache/salt/master/git_pillar/work/6UMN6PLOTjtzIOfVZQddUm_lEiq5dnEwwtfYWjzDVfo=/main/update.lk) which does not exist, exception : [Errno 2] No such file or directory: '/var/cache/salt/master/git_pillar/work/6... repohash removed ...fo=/main/update.lk'
[DEBUG ] The functions from module 'local_cache' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded local_cache.prep_jid
[DEBUG ] The functions from module 'localfs_key' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded localfs_key.list_all
[DEBUG ] resource_registry: derived-index rebuild 0.000s (0 types, 0 minions, version=(None, 0))
[DEBUG ] The functions from module 'localfs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded localfs.list
[DEBUG ] Adding minions for job 20260907160900315806: []
[DEBUG ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded nested.output
[TRACE ] data = {'__env__ ssh://git@gitlab-internal/repo-pillar.git': True}
__env__ ssh://git@gitlab-internal/repo-pillar.git:
True
[INFO ] Runner completed: 20260907160900315806
[DEBUG ] Closing _TCPPubServerPublisher instance
[DEBUG ] Runner return: {'fun': 'runner.git_pillar.update', 'jid': '20260907160900315806', 'user': 'salt', 'fun_args': [], '_stamp': '2026-09-07T16:09:01.188308+00:00', 'return': {'__env__Source: saltstack/salt