Huge latency in retransmitted RTMP stream

[Yu You]

Usually, the default attributes of the x265enc (and H264 encoder) are not
ideally suitable for real-time use cases. Try something, e.g. fixed GOP
(key-int-max=30), speed-preset=superfast tune=zerolatency etc. In addition,
* rtph265pay aggregate-mode=aggregate-mode;
* use stream-format=byte-stream;
* use "h265parse config-interval=-1";
* add queue before and after encoder;

Alternatively, try to use HW encoders like NVCodec provided if using NVidia
cards.

Good luck,

Yu

On Thu, 18 Aug 2022 at 15:33, robxkx7 via gstreamer-devel <
gstreamer-devel at lists.freedesktop.org> wrote:

> Dear all,
>
> I have a quite complex streaming architecture where:
> 1. an analog infrared camera is connected to a custom board based on a
> Jetson TX2i
> 2. the board uses the Jetson's hw H.265 encoder and transmits the images
> via LTE as UDP stream
> 3. a VM receives the UDP stream, re-encodes it as H.264 and publishes a
> RTMP stream using NGINX (let's call it RTMP-VM)
> 4. clients connect to this RTMP stream and decode it locally.
>
> The purpose of the whole pipeline is to stream with the lowest latency and
> with the lowest occupied LTE bandwidth possible (for the step (2) only, (3)
> and (4) rely on high-speed connections) to multiple clients (which should
> be able to decode it easily --- that is, with a traditional player or, even
> better, youtube-style). I am therefore open to other alternatives, as long
> as they fit the two requirements above (and are supported by the gstreamer
> version on the board).
> The gstreamer version on the Jetson-based board is quite old (1.14 IIRC
> and, unfortunately --- due to the atrocious support by NVIDIA --- no way to
> upgrade it), the other machines are standard so I have a recent version.
>
> I have tried streaming, and the latency is simply HUGE --- something in
> the order of 10 to 15 seconds !!
> Network latency, both for the uplink LTE link and the downlink wired link,
> is in the order of 20-30 ms, and the pipelines (measured with
> GST_DEBUG="GST_TRACER:7" GST_TRACERS=latency) take roughly 35 ms for the
> encoding+UDP streaming on the Jetson, and between 10 and 30 ms for the
> reencoding and RTMP retransmission, therefore I cannot explain myself where
> this latency originates.
> If, instead of doing this re-encoding+RTMP streaming, I simply send the
> UDP packets to the destination where they are decoded, the latency is in
> the order of 100-150 ms (which is acceptable for my application) --- but I
> have to create a RTMP stream where "random" users can connect, thus this
> (=multiudpsink) is not an option.
> Also, please note that I experience the same latency even when doing a
> test with my PC streaming a low-res (256x144 pixels) video from my webcam
> to the RTMP-VM, getting it back locally for viewing.
>
> To allow people reproducing my setup (and since I have no access to the
> embedded board until next Tuesday), I give here the pipelines for the PC
> streaming test (since the behaviour is exactly the same).
> The resolution of the stream from the embedded board is slightly higher
> (720x288 pixels --- the occupied bw max out at roughly 1.3 Mbps) but this
> does not seem to be the main issue. Also I limit the MTU in rtph265pay
> since data passes through a VPN (and with this setting I avoid packet
> fragmentation), but it does not seem to impact the latency in any way.
>
> IP of the RTMP-VM: 10.0.0.2
>
> A. video encoding and upstream on my PC:
> gst-launch-1.0 v4l2src ! jpegdec ! videoscale !
> video/x-raw,width=256,height=144 ! x265enc ! rtph265pay mtu=970 ! udpsink
> host=10.0.0.2 port=5000
>
> B. re-encoding and RTMP transmission on the RTMP VM:
> gst-launch-1.0 udpsrc port=5000 ! queue min-threshold-time=20000000
> max-size-time=100000000 ! "application/x-rtp, encoding-name=H265,
> payload=96" ! rtph265depay ! h265parse ! avdec_h265 ! videoconvert !
> x264enc sliced-threads=true tune=zerolatency ! h264parse ! flvmux
> streamable=true ! queue max-size-time=0 ! rtmpsink location='rtmp://
> 127.0.0.1/live/stream live=true'
>
> C. local decoding of the RTMP stream on my PC:
> gst-launch-1.0 rtmpsrc location="rtmp://10.0.0.2/live/stream" ! decodebin
> ! autovideosink
>
> If I use
> gst-launch-1.0 uridecodebin uri="rtmp://10.0.0.2/live/stream"
> buffer-duration=50000000 ! autovideosink
> for (C), the pipeline gets paused all the time, printing on the terminal:
>
> Buffering, setting pipeline to PAUSED ...
> Done buffering, setting pipeline to PLAYING ...
>
> (I don't know if this is normal, but I expect that if data was coming at
> full speed, the pipeline wouldn't have been paused...).
>
> I have literally spent days attempting every approach I have found on the
> web and tweaking every single component of the pipelines, but to no avail.
> Also, the fact that I cannot identify where this latency source lies simply
> drives me nuts...
> Can anyone help me, please?
> Thanks a lot in advance !
>
> Best,
> Rob
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20220818/f79d9ef5/attachment-0001.htm>