Huge latency in retransmitted RTMP stream

[robxkx7]

Dear all,

I have a quite complex streaming architecture where:
1. an analog infrared camera is connected to a custom board based on a Jetson TX2i
2. the board uses the Jetson's hw H.265 encoder and transmits the images via LTE as UDP stream
3. a VM receives the UDP stream, re-encodes it as H.264 and publishes a RTMP stream using NGINX (let's call it RTMP-VM)
4. clients connect to this RTMP stream and decode it locally.

The purpose of the whole pipeline is to stream with the lowest latency and with the lowest occupied LTE bandwidth possible (for the step (2) only, (3) and (4) rely on high-speed connections) to multiple clients (which should be able to decode it easily --- that is, with a traditional player or, even better, youtube-style). I am therefore open to other alternatives, as long as they fit the two requirements above (and are supported by the gstreamer version on the board).
The gstreamer version on the Jetson-based board is quite old (1.14 IIRC and, unfortunately --- due to the atrocious support by NVIDIA --- no way to upgrade it), the other machines are standard so I have a recent version.

I have tried streaming, and the latency is simply HUGE --- something in the order of 10 to 15 seconds !!
Network latency, both for the uplink LTE link and the downlink wired link, is in the order of 20-30 ms, and the pipelines (measured with GST_DEBUG="GST_TRACER:7" GST_TRACERS=latency) take roughly 35 ms for the encoding+UDP streaming on the Jetson, and between 10 and 30 ms for the reencoding and RTMP retransmission, therefore I cannot explain myself where this latency originates.
If, instead of doing this re-encoding+RTMP streaming, I simply send the UDP packets to the destination where they are decoded, the latency is in the order of 100-150 ms (which is acceptable for my application) --- but I have to create a RTMP stream where "random" users can connect, thus this (=multiudpsink) is not an option.
Also, please note that I experience the same latency even when doing a test with my PC streaming a low-res (256x144 pixels) video from my webcam to the RTMP-VM, getting it back locally for viewing.

To allow people reproducing my setup (and since I have no access to the embedded board until next Tuesday), I give here the pipelines for the PC streaming test (since the behaviour is exactly the same).
The resolution of the stream from the embedded board is slightly higher (720x288 pixels --- the occupied bw max out at roughly 1.3 Mbps) but this does not seem to be the main issue. Also I limit the MTU in rtph265pay since data passes through a VPN (and with this setting I avoid packet fragmentation), but it does not seem to impact the latency in any way.

IP of the RTMP-VM: 10.0.0.2

A. video encoding and upstream on my PC:
gst-launch-1.0 v4l2src ! jpegdec ! videoscale ! video/x-raw,width=256,height=144 ! x265enc ! rtph265pay mtu=970 ! udpsink host=10.0.0.2 port=5000

B. re-encoding and RTMP transmission on the RTMP VM:
gst-launch-1.0 udpsrc port=5000 ! queue min-threshold-time=20000000 max-size-time=100000000 ! "application/x-rtp, encoding-name=H265, payload=96" ! rtph265depay ! h265parse ! avdec_h265 ! videoconvert ! x264enc sliced-threads=true tune=zerolatency ! h264parse ! flvmux streamable=true ! queue max-size-time=0 ! rtmpsink location='rtmp://127.0.0.1/live/stream live=true'

C. local decoding of the RTMP stream on my PC:
gst-launch-1.0 rtmpsrc location="rtmp://10.0.0.2/live/stream" ! decodebin ! autovideosink

If I use
gst-launch-1.0 uridecodebin uri="rtmp://10.0.0.2/live/stream" buffer-duration=50000000 ! autovideosink
for (C), the pipeline gets paused all the time, printing on the terminal:

Buffering, setting pipeline to PAUSED ...
Done buffering, setting pipeline to PLAYING ...

(I don't know if this is normal, but I expect that if data was coming at full speed, the pipeline wouldn't have been paused...).

I have literally spent days attempting every approach I have found on the web and tweaking every single component of the pipelines, but to no avail. Also, the fact that I cannot identify where this latency source lies simply drives me nuts...
Can anyone help me, please?
Thanks a lot in advance !

Best,
Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20220818/dd74fb33/attachment.htm>