New Background mode option: temporal interpolation

Is there an existing feature request for this?

• I have searched the existing requests

Description

Hello,

Is there a plan to implement the equivalent of the "Use previous and future frames to fill borders" + "Soft borders" options of Deshaker (VirtualDub plugin) in Gyroflow ? I guess this would be a new "Background mode" option, or a complementary option which could work in conjunction with existing background modes.

This feature is like a temporal interpolation/blending which recreates the missing pixels at the edge of the stabilized image, based on the nearest (in term of time: backward and/or forward) know pixel at these locations according to camera motions. Several past or future frames can be used to regenerate the missing pixels, blending them together with weight based on their temporal distance. This feature is very powerful when the camera is shaky and/or rotates. In lot of cases it makes it possible to avoid cropping/zooming when performing stabilization and it's way better than black borders or mirroring/repeating edge pixels.

Here are the descriptions of these 2 options in Deshaker for reference:

Use previous and future frames to fill borders

This is a kind of additional edge compensation. When the current frame doesn't contain any image data in the "real world area" we want in our output, it can search in past and future frames and use that image data instead. As long as that area is part of the object (or background) that was stabilized on in pass 1, this usually works perfectly. But when other objects move in these areas, it can look a bit strange. It's usually better than black anyway, so I strongly recommend using this feature. Just make sure you crop any borders from the source clip before Deshaker. Otherwise this won't work very well at all. If the current frame is number 50, it searches frames in the following order to find the image data it needs: 50, 49, 51, 48, 52, 47, 53, 46, 54 etc. So for each pixel the closest frame (chronologically) is always used. You can set how many frames to store internally for this feature. More is always better but keep in mind that each frame needs up to 8 MB of memory (for full HD) and it can take a long time to search through all frames for image data. (To be able to use more memory, you can either use 64-bit Deshaker, or use the 4GB patch on the 32-bit VirtualDub.exe file.)

Try using this feature on a clip with lots of panning, and zoom out by setting Extra zoom factor to something like 0.6. Then you'll get a panorama like effect. It's slow, but fun. :)

Soft borders

If enabled, all relevant previous and future frames that Deshaker has stored in memory will be used to calculate the border (as opposed to just using the closest frame in time). For each pixel, it takes an average of the corresponding pixels in all these frames, weighted so that the frames closest to the current frame (in time) get higher importance.

This option will also allow you to make the transitions between a frame and the frame(s) beneath it smoother. The transition width value you supply is how far in from the edge of a frame the transition will start. The transition will be applied also to the current frame, so you'll actually lose some information in the edges of the main frame by using this option. But I'd say it's worth it. If you don't like it, just set the transition width to 0.

Enabling Soft borders improves the appearance of the border quite a lot, especially when there are objects moving there, but it is also very slow when there are lots of border pixels.