TestRustcVersionIdentification downloads FLIRT signatures at test time and fails when the download does

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Two tests in tests/analyses/decompiler/test_rust_decompiler.py::TestRustcVersionIdentification download FLIRT signatures over the network at run time, and fail when that download fails rather than when anything about angr is wrong.

Both failed in Test with coverage run 34119042125 attempt 2, commit 26e7e51d, in two different shards:

TestRustcVersionIdentification::test_default_sig_dir            results-6/junit.xml, 0 s
  AssertionError: False is not true : get_default_sig_dir() returned None

TestRustcVersionIdentification::test_fmt_version_nightly_2023_05_22_O3   results-10/junit.xml, 31 s
  AssertionError: None != '1.71.0' : fmt [nightly-2023-05-22-O3]: expected 1.71.0, got None

Mechanism

get_default_sig_dir() returns None unless a sentinel file already exists in the platform cache directory or _download_flirt_signatures() succeeds, and that helper fetches flirt_signatures/archive/refs/heads/master.tar.gz and turns every failure into a warning:

    except Exception as e:  # pylint:disable=broad-exception-caught
        l.warning("Failed to download FLIRT signatures from %s: %s", TARBALL_URL, e)
        shutil.rmtree(staging, ignore_errors=True)
        return False

So a network error, an HTTP error or a rate limit becomes sig_dir = None. test_default_sig_dir then fails on its own assertion, and test_fmt_version_nightly_2023_05_22_O3 fails because RustcVersionIdentification has no signatures to match and leaves rustc_version as None. The 0-second duration of the first is the tell: nothing was downloaded before it gave up.

Both pass locally in 13.7 s on a machine whose cache is already populated, which is why this only shows up on fresh runners.

It does not surface in the workflow's own conclusion

Run 34119042125's conclusion is success. Only the Test Results check published by the Report job is failure, because the shards upload their reports and exit 0. Anyone checking the run rather than that check will not see these at all.

Suggested direction

A test that fetches its input at run time fails for reasons that have nothing to do with what it asserts. Either prime the cache in a workflow step, so a download failure shows up as a setup failure with a diagnosable error, or skip these when get_default_sig_dir() returns None rather than asserting on it.

Happy to send the patch, whichever you prefer.