Testing probot app significantly slower in v13

Bug Report

Testing my probot app seems to have gotten significantly slower in probot v13.

I noticed when upgrading from probot v12 to v13, that my unit tests were taking nearly 10x longer to run (from ~5s to ~50s in our internal app). I though this could be due to swapping from nock to msw for mocking github API requests, but after reproducing the issue using just msw, I think the culprit is a change introduced in probot v13.

NOTE: I have not had a chance to test the real-world execution time of my app with probot v13, as I've been more concerned with the test time. As our internal app is not a critical app, we might just test it anyway and roll back if there are any issues. I will update this issue if we end up doing this and discovering anything.

Current Behavior

Tests run much slower in probot v13 compared to v12.

I have reproduced the issue in an example probot app that I created using npx create-probot-app my-first-app.

Reproduction repo

Steps to reproduce:

git clone https://github.com/askoufis/test-probot-app
cd test-probot-app
pnpm install
pnpm run test

The master version of the app is using probot v12 and msw. I replaced nock with msw because nock doesn't work with probot v13 since it now uses native fetch, and I didn't want to have both the probot version and msw migration influencing the reproduction.

You should observe that the 2 tests complete in ~1s (<1s on my M1 pro). Adding more tests does not significantly influence the test run time; each test only adds <100 ms.

Next steps:

git checkout probot-13
pnpm install
pnpm run test

This version of the app updates probot to 13.0.2, and that's it.

You should observe that the tests take significantly longer, ~4s on my machine. Adding more tests increases the test time by ~3s per test added.

Expected behavior/code Test time should ideally be somewhat similar between major versions.

Environment

• Probot version(s): v13.0.2
• Node/npm version: 20.11.0
• OS: macOS 14.2.1

Possible Solution

Doing some rudimentary console.log timing, I can't see anything during the actual probot app execution that taking up lots of time. Interestingly, if you delete all but 1 test, it runs in ~1s, and jest indicates that it runs in <100ms on my machine. This leads me to believe that the culprit is potentially related to some kind of cleanup step that the probot app is doing.