`INP001` doesn't respect caching behavior

See #2075. It looks like the underlying issue (that we don't invalidate the cache when an __init__.py is added or removed) is back with the latest cache updates.

Originally posted by @matthias-roussel-soyhuce in https://github.com/astral-sh/ruff/issues/2075#issuecomment-1612618794