QUIC Transport architecture for GStreamer

[Samuel Hurst]

Hello all,

TL;DR: How best to implement a bidirectional transport src/sink in 
GStreamer?

I've been looking at media transport over QUIC transport [1] for quite 
some time, and I would like to start implementing some things in 
GStreamer, however I'm unsure as to exactly what architecture I should 
go with and would appreciate some advice from some GStreamer experts.

My initial thoughts involved writing a quicsrc/quicsink element that 
would itself sit between the rest of the pipeline and a pair of udpsrc 
and dynudpsink elements. QUIC Datagrams can be passed on sometimes pads, 
with more request pads created dynamically for each QUIC stream and 
announced via named signals.

In the example below, an rtpquicdemux element would manage the QUIC 
datagram and stream flows to enable reception of RTP over QUIC [2], 
passing regular RTP packets down to existing GStreamer RTP elements:

+--------+  +---------+    +---------+
| udpsrc +-->         |    |         | +-------------+ +-----------+
+--------+  |         |    |         +-> rtp...depay +-> decodebin +->
             | quicsrc +----> rtpquic | +-------------+ +-----------+
+--------+  |         +---->  demux  | +-------------+ +-----------+
| dynudp <--+         |    |         +-> rtp...depay +-> decodebin +->
|  sink  |  |         |    |         | +-------------+ +-----------+
+--------+  +---------+    +---------+

However, in the case where you need to send and receive on the same QUIC 
transport connection, this model seems to conceptually break down, as 
the quicsrc and quicsink elements wouldn't be able to share the QUIC 
transport connection, which doesn't make sense for a bidirectional media 
session such as a video call:

+--------+  +---------+        +----------+ +--------+
| udpsrc +-->         |        |          +-> dynudp |
+--------+  |         +->      |          | |  sink  |
             | quicsrc |  ... --> quicsink | +--------+
+--------+  |         +->      |          | +--------+
| dynudp <--+         |      -->          <-+ udpsrc |
|  sink  |  |         |        |          | +--------+
+--------+  +---------+        +----------+

In addition, any bidirectional streams would need sink pads on the 
quicsrc element to pass messages back, which sounds quite messy to me.

Instead, I wondered whether or not trying to incorporate the udpsrc and 
dynudpsink elements was just over-complicating it for myself, and 
instead I should just have the quicsrc and quicsink elements be 
standalone, with a shared QUIC library (not element, but difficult to 
show the difference in asciiart) between them managing the socket and 
the QUIC transport connection transparently?

                         +---------+
                         | sockets |
                         +----^----+
                              |
                         +----v----+
      +------------------> quiclib <-------------------+
      |                  +---------+                   |
+----v----+  +-------+ +-----+ +------+ +-----+       |
|         +-->rtpquic+-> rtp +->decode+->audio|  +----v-----+
|         |  | demux | |depay| | bin  | |sink |  |          |
| quicsrc |  +-----+-+ +----++ +--+-+-+ +---+-+  |          |
|         |    +-----+ +-+----+ ++--+ +-+-+---+  | quicsink |
|         |    |audio+->encode+->rtp+->rtpquic+-->          |
+---------+    | src | | bin  | |pay| |  mux  |  |          |
                +-----+ +------+ +---+ +-------+  +----------+

The above architecture is what happens in things like the httpsrc 
elements which own the socket internally. I'd imagine the "quiclib" 
would effectively be a singleton instance that would mediate all QUIC 
transport connections in a given pipeline, in case there are multiple 
sources/sinks, such as in an RTP mixer. However, this has challenges of 
its own in terms of which element "owns" each connection, but that's a 
problem for later on.

What would you suggest is most inkeeping with the intended GStreamer 
design ethos, and isn't going to risk me tripping myself up down the line?

Best regards,
Sam

--
[1]: https://www.rfc-editor.org/rfc/rfc9000.html
[2]: https://datatracker.ietf.org/doc/draft-ietf-avtcore-rtp-over-quic/