webcodec, websocket, and webtransport

hi! me and my s/o use neko every day and we love it. it's awesome and im super grateful for all the work that's been put in.

i host neko from a machine in my home network (so i can take advantage of an igpu). all the stuff around webrtc (ICE/TURN) is buggy and i often have to just restart the neko server to get things to work every once in a while. i also have to add a special firewall rule to get the udp routed properly to the docker container the way i have my network setup.

i saw that it was planned to add more streaming backends. i saw #508 as an initial attempt for websocket, and thought I would try my hand at this task using your feedback in that thread. i am also big fan of webtransport and have been playing with it since the first draft, and wanted to implement this for neko, as it is standardized, implemented, and is the perfect fit for the use case

my implementation uses webcodecs instead of fmp4/mse, as latency is much better. even after tuning mse to 20ms, webcodec still felt better. webcodec also has the advantage of being one path to support both vp8 and h264

full disclaimer - every line of code is llm generated or edited. this issue of course is written by hand, and i have also reviewed and iterated over every line of all three prs, also by hand. that said i have not yet taken the time to refine comments and match up style with more reading of conventions of existing code.

i did this in three prs on top of each other, to separate the features to make them hopefully easier to look at and merge and fix up one step at a time.

1. https://github.com/tuxpainter/neko/pull/1 changes the media subscriptions path, which makes the streaming backend not webrtc specific, and exposes the timing information from gstreamer.

2. https://github.com/tuxpainter/neko/pull/2 websocket and webcodec support. we send bytes over websocket and handle in the vue client. access http://localhost:3001/?media=webcodecs-ws to use the websocket version.

i've tested this locally with vp8 and it seems to work. i've also used the legacy websocket to create a mouse/kb control api to completely remove the webrtc dependency. i use the existing gstreamer pipeline, and we should properly support both vp8 and h264 (i still need to test h264)

1. https://github.com/tuxpainter/neko/pull/3 webtransport support. this is a big pr. webtransport runs over http3/quic, and quic requires tls 1.3, so this is gated behind providing tls certificates for the server.

webtransport provides a major advantage of being integrated with the streams api, which means that using it, we get "free" backpressure handling, which does not exist in the websocket backend.

i'm continuing to use the websocket based mouse/kb control api, but it may be worth in the future using webtransport for the mouse, as webtransport supports unreliable transport mode.

i've test this locally as well with vp8. this is an awful pain to test. webtransport does not obey local certificate exceptions, so some special code is needed. this is the furthest from mergability because of this. caddy will support webtransport on the next major release, so users running caddy in front of neko could reuse the caddy obtained certificate and proxy through it as a very easy deploy path.

my long term vision with this is to eventually also support MOQ, which is not yet finalized, but is effectively a standard for webcodec compatible payloads over webtransport which can be relayed (cf for example hosts edge moq gateways), which provides the bandwidth savings that webrtc provides. there is a go implementation in active development

let me know what work might be needed to start getting these in, any changes, comments, feedback, etc. i know that the vue frontend is legacy, but i did not want to over reach and also write a react frontend for these flows. that said i do write some react for , and would be open to doing this if its something you would want before getting websocket and/or webtransport in, though it's not my favourite.