Scroll lag over remote ssh with IME enabled

Describe the bug I have been using neovide with a remote neovim server connected through SSH port-forwarding for over a year. I recently upgraded neovide from 0.15.2 to 0.16.2 and immediately noticed that scrolling got very laggy - both in terms of the scroll speed and how quickly it responds to my mouse input. I bisected between 0.15.2 and 0.16.2 found that https://github.com/neovide/neovide/pull/3221 is the culprit.

Once I disable IME :let g:neovide_input_ime = v:false, the scrolling is smooth once again.

To Reproduce I have been unable to reproduce this on a local file. It only happens over a remote ssh setup. Probably https://github.com/neovide/neovide/pull/3221 changed something that made neovide very sensitive to the latency to nvim server.

Desktop (please complete the following information):

• OS: Debian Linux (both local and remote)
• Neovide Version: 0.16.2 (also reproducible on the latest commit on the main branch)
• Neovim Version: 0.12.4

Happy to provide more info if needed.