Huge latency in retransmitted RTMP stream

[robxkx7]

Dear Yu,

Thanks for your reply, I wasn't aware of some of the options you've set and I'll gladly integrate them.
Actually, I am using the hw encoders on the board, the command I gave in my message was only to allow people to reproduce my problem without them.
In particular, on the board I use a C++ code I've written which does the following (+- some options of the different components, since I've tried tweaking everything I don't have a "standard" pipeline anymore.... but this should give you a rough idea):

gst-launch-1.0 v4l2src device="/dev/video0" ! video/x-raw,width=720,height=288,format=YUY2 ! videoconvert ! "video/x-raw, format=GRAY8" ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420" ! nvv4l2h265enc ! rtph265pay ! udpsink port=5000 host=10.0.0.2

When I use point-to-point communication with UDP (that is, I encode on the board and decode+display on the UDP sink side), latency is just fine (roughly 100-150 ms, depending on network delays, which is fine for my application). However, when I add the intermediate step that does the RTMP retransmission, latency skyrockets to 10-15 seconds :(

Any clue about where this buffering is occurring?

Thanks a lot in advance and have a nice day!

Best,
Rob
------- Original Message -------
On Thursday, August 18th, 2022 at 4:37 PM, Yu You <youyu.youyu at gmail.com> wrote:

> Usually, the default attributes of the x265enc (and H264 encoder) are not ideally suitable for real-time use cases. Try something, e.g. fixed GOP (key-int-max=30), speed-preset=superfast tune=zerolatency etc. In addition,
> * rtph265pay aggregate-mode=aggregate-mode;
>
> * use stream-format=byte-stream;
>
> * use "h265parse config-interval=-1";
> * add queue before and after encoder;
>
> Alternatively, try to use HW encoders like NVCodec provided if using NVidia cards.
>
> Good luck,
>
> Yu
>
> On Thu, 18 Aug 2022 at 15:33, robxkx7 via gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>
>> Dear all,
>>
>> I have a quite complex streaming architecture where:
>> 1. an analog infrared camera is connected to a custom board based on a Jetson TX2i
>> 2. the board uses the Jetson's hw H.265 encoder and transmits the images via LTE as UDP stream
>> 3. a VM receives the UDP stream, re-encodes it as H.264 and publishes a RTMP stream using NGINX (let's call it RTMP-VM)
>> 4. clients connect to this RTMP stream and decode it locally.
>>
>> The purpose of the whole pipeline is to stream with the lowest latency and with the lowest occupied LTE bandwidth possible (for the step (2) only, (3) and (4) rely on high-speed connections) to multiple clients (which should be able to decode it easily --- that is, with a traditional player or, even better, youtube-style). I am therefore open to other alternatives, as long as they fit the two requirements above (and are supported by the gstreamer version on the board).
>> The gstreamer version on the Jetson-based board is quite old (1.14 IIRC and, unfortunately --- due to the atrocious support by NVIDIA --- no way to upgrade it), the other machines are standard so I have a recent version.
>>
>> I have tried streaming, and the latency is simply HUGE --- something in the order of 10 to 15 seconds !!
>> Network latency, both for the uplink LTE link and the downlink wired link, is in the order of 20-30 ms, and the pipelines (measured with GST_DEBUG="GST_TRACER:7" GST_TRACERS=latency) take roughly 35 ms for the encoding+UDP streaming on the Jetson, and between 10 and 30 ms for the reencoding and RTMP retransmission, therefore I cannot explain myself where this latency originates.
>> If, instead of doing this re-encoding+RTMP streaming, I simply send the UDP packets to the destination where they are decoded, the latency is in the order of 100-150 ms (which is acceptable for my application) --- but I have to create a RTMP stream where "random" users can connect, thus this (=multiudpsink) is not an option.
>> Also, please note that I experience the same latency even when doing a test with my PC streaming a low-res (256x144 pixels) video from my webcam to the RTMP-VM, getting it back locally for viewing.
>>
>> To allow people reproducing my setup (and since I have no access to the embedded board until next Tuesday), I give here the pipelines for the PC streaming test (since the behaviour is exactly the same).
>> The resolution of the stream from the embedded board is slightly higher (720x288 pixels --- the occupied bw max out at roughly 1.3 Mbps) but this does not seem to be the main issue. Also I limit the MTU in rtph265pay since data passes through a VPN (and with this setting I avoid packet fragmentation), but it does not seem to impact the latency in any way.
>>
>> IP of the RTMP-VM: 10.0.0.2
>>
>> A. video encoding and upstream on my PC:
>> gst-launch-1.0 v4l2src ! jpegdec ! videoscale ! video/x-raw,width=256,height=144 ! x265enc ! rtph265pay mtu=970 ! udpsink host=10.0.0.2 port=5000
>>
>> B. re-encoding and RTMP transmission on the RTMP VM:
>> gst-launch-1.0 udpsrc port=5000 ! queue min-threshold-time=20000000 max-size-time=100000000 ! "application/x-rtp, encoding-name=H265, payload=96" ! rtph265depay ! h265parse ! avdec_h265 ! videoconvert ! x264enc sliced-threads=true tune=zerolatency ! h264parse ! flvmux streamable=true ! queue max-size-time=0 ! rtmpsink location='rtmp://127.0.0.1/live/stream live=true'
>>
>> C. local decoding of the RTMP stream on my PC:
>> gst-launch-1.0 rtmpsrc location="rtmp://10.0.0.2/live/stream" ! decodebin ! autovideosink
>>
>> If I use
>> gst-launch-1.0 uridecodebin uri="rtmp://10.0.0.2/live/stream" buffer-duration=50000000 ! autovideosink
>> for (C), the pipeline gets paused all the time, printing on the terminal:
>>
>> Buffering, setting pipeline to PAUSED ...
>> Done buffering, setting pipeline to PLAYING ...
>>
>> (I don't know if this is normal, but I expect that if data was coming at full speed, the pipeline wouldn't have been paused...).
>>
>> I have literally spent days attempting every approach I have found on the web and tweaking every single component of the pipelines, but to no avail. Also, the fact that I cannot identify where this latency source lies simply drives me nuts...
>> Can anyone help me, please?
>> Thanks a lot in advance !
>>
>> Best,
>> Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20220819/c56f59cd/attachment.htm>