Prevent `scripts/run_all.py` from bit-rotting

In #4146, I had to make a bunch of fixes to `run_all.py` to have a successful run. We should somehow try to avoid this script from bit-rotting.