FEC and retransmissions for video

[Mathieu Duponchelle]

Hi Lauri,



On 05/14/2018 02:53 PM, Lauri Ehrenpreis wrote:
> Hi!
>
> I tried to set up a pipeline which does both retransmissions and FEC for video. My hope was that first pipeline will try to recover packet with FEC and if that fails it will request retransmission. As there was no working example for such setup I used gst-plugins-good/tests/examples/rtp/client-rtpaux.c and gst-plugins-good/tests/examples/rtp/server-rtpaux.c as basis and added request-fec-decoder & request-fec-encoder signal handlers there. 
>
> After adding FEC handlers gstrtpbin came up with following pipeline on client side (attached image). So retransmission component is before jitterbuffer and FEC is after. Do I understand correctly the pipeline will first try to request retransmission if packet is lost and only after the retransmission did not deliver the packet in time for jitterbuffer, it will try use FEC to recover packet? Why it is done this way - wouldn't it be better to try FEC first?

Yes, you are correct, FEC recovery will only be attempted once a packet is actually
considered lost.

While we could indeed reconstruct packets as soon as they are late, there is no
guarantee that the reconstructed packet would be "complete", at least with ULPFEC.

As you might already know, ULP stands for "uneven level protection", and works based
on the assumption that with most video codecs / payloaders, the most important part
of a packet is at the start, which means that a percentage of protection can be applied
at the packet level by only protecting the initial portion, possibly making exceptions
for keyframe packets for example, which could be fully protected.

This means that while we could reconstruct a packet, if retransmission has been
enabled it makes sense to ask for that packet to be retransmitted anyway.

I hope that made sense :)

-- 
Mathieu Duponchelle · https://www.centricular.com