FEC and retransmissions for video

[Lauri Ehrenpreis]

ULP does not always protect only part of the packets. It can also protect
full packets. In fact the current implementation of gstrtpulpfecenc.c is
only protecting full packets. Also the fec decored knows if it recovered
full packet or only part of the pecket.

Given this I see no reason why FEC decoder should be placed downstream from
jitterbuffer. Being downstream and only working when jitterbuffer reported
loss means that we are not getting any latency benefit from FEC as it only
recovers packet _after_ jitterbuffer timed out the packet and reported
loss. Jitterbuffer latency can be very large - 2 sec for example. This
means that even if we paid for the bandwidth and sent FEC, we still get 2
sec delay. Also additional bandwidth will be wasted because RTX will anyway
retransmit the packet which was already recovered by FEC. So double loss
because of current implementation.

To make RTX and FEC work together I placed FEC decoder upstream from
jitterbuffer (using request-aux-receiver which returns a bin containing
rtprtxreceive, rtpstorage and rtpulpfecdec). Also I modified FEC decoder so
that it looks for the sequence number gaps in storage on every incoming
packet and tries to recover those immediately. This way FEC will reduce the
latency if it recovers something and also it can work together with rtx.
There are minor optimizations to be done since for example currently
jitterbuffer also requests retransmit for FEC packets. These requests
should be ignored on sender side.

I think in case it's not known if FEC will be able to recover full or only
partial packets, there should be a FEC decoder before jitterbuffer which
only recovers full packets and another one downstream which recovers
partial packets when jitterbuffer considers those lost.

--
LauriE

On Wed, May 16, 2018 at 3:43 PM, Mathieu Duponchelle <
mathieu at centricular.com> wrote:

> Hi Lauri,
>
>
>
> On 05/14/2018 02:53 PM, Lauri Ehrenpreis wrote:
> > Hi!
> >
> > I tried to set up a pipeline which does both retransmissions and FEC for
> video. My hope was that first pipeline will try to recover packet with FEC
> and if that fails it will request retransmission. As there was no working
> example for such setup I used gst-plugins-good/tests/examples/rtp/client-rtpaux.c
> and gst-plugins-good/tests/examples/rtp/server-rtpaux.c as basis and
> added request-fec-decoder & request-fec-encoder signal handlers there.
> >
> > After adding FEC handlers gstrtpbin came up with following pipeline on
> client side (attached image). So retransmission component is before
> jitterbuffer and FEC is after. Do I understand correctly the pipeline will
> first try to request retransmission if packet is lost and only after the
> retransmission did not deliver the packet in time for jitterbuffer, it will
> try use FEC to recover packet? Why it is done this way - wouldn't it be
> better to try FEC first?
>
> Yes, you are correct, FEC recovery will only be attempted once a packet is
> actually
> considered lost.
>
> While we could indeed reconstruct packets as soon as they are late, there
> is no
> guarantee that the reconstructed packet would be "complete", at least with
> ULPFEC.
>
> As you might already know, ULP stands for "uneven level protection", and
> works based
> on the assumption that with most video codecs / payloaders, the most
> important part
> of a packet is at the start, which means that a percentage of protection
> can be applied
> at the packet level by only protecting the initial portion, possibly
> making exceptions
> for keyframe packets for example, which could be fully protected.
>
> This means that while we could reconstruct a packet, if retransmission has
> been
> enabled it makes sense to ask for that packet to be retransmitted anyway.
>
> I hope that made sense :)
>
> --
> Mathieu Duponchelle · https://www.centricular.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180518/21c24b98/attachment.html>