HF Detector loading process sometimes times out, then crashes the run
Author: leondzCreated Aug 13, 2024Updated Sep 13, 2026
Labelsbugdetectorsuxneeds-triage
Sometimes there's a delay when moving to a new probe, then the run quits. My hypothesis is that this happens when HF is down, even if the model has been used before (seems that HF model load might be checking their server). Need more work on reproducing this; would like this to fail more gracefully, e.g. by skipping just this one detector (or this one probe if it only has one detector defined, which is the failing one).
log trace:
2024-08-13 17:42:19,855 DEBUG harness: probe list iteration completed
2024-08-13 17:42:19,855 INFO probe init: <garak.probes.lmrc.Bullying object at 0x775754d94510>
2024-08-13 17:42:19,855 DEBUG deprecation warning - probe probes.lmrc.Bullying using recommend_detector instead of primary_detector
2024-08-13 17:42:19,883 INFO detector init: <garak.detectors.toxicity.ToxicCommentModel object at 0x775754d97ed0>
2024-08-13 17:42:19,883 DEBUG Using cpu, based on torch environment evaluation
2024-08-13 17:42:19,884 DEBUG Starting new HTTPS connection (1): huggingface.co:443
2024-08-13 17:42:29,962 DEBUG Starting new HTTPS connection (2): huggingface.co:443
2024-08-13 17:42:40,008 WARNING error instantiating module <module 'garak.detectors.toxicity' from '/home/lderczynski/anaconda3/envs/garak-test/lib/python3.11/site-packages/garak/detectors/toxicity.py'> class ToxicCommentModel
2024-08-13 17:42:40,009 ERROR detector load failed: toxicity.ToxicCommentModel, skipping >>
2024-08-13 17:42:40,009 INFO harness init: <garak.harnesses.base.Harness object at 0x7756904d97d0>
2024-08-13 17:42:40,009 WARNING No detectors, nothing to do
2024-08-13 17:42:40,009 ERROR No detectors, nothing to do
Traceback (most recent call last):
File "/home/lderczynski/anaconda3/envs/garak-test/lib/python3.11/site-packages/garak/cli.py", line 502, in main
command.probewise_run(
File "/home/lderczynski/anaconda3/envs/garak-test/lib/python3.11/site-packages/garak/command.py", line 219, in probewise_run
probewise_h.run(generator, probe_names, evaluator, buffs)
File "/home/lderczynski/anaconda3/envs/garak-test/lib/python3.11/site-packages/garak/harnesses/probewise.py", line 107, in run
h.run(model, [probe], detectors, evaluator, announce_probe=False)
File "/home/lderczynski/anaconda3/envs/garak-test/lib/python3.11/site-packages/garak/harnesses/base.py", line 86, in run
raise ValueError(msg)
ValueError: No detectors, nothing to doSource: NVIDIA/garak