kmssink and ultra low latency

[Nicolas Dufresne]

Le lundi 21 novembre 2022 à 13:24 +0000, Peter Allen via gstreamer-devel a
écrit :
> I have an application that needs ultra-low latency decoding and display of a
> 720p/60 H264 rtp stream. I'm using Allwinner H3 (currently an Orange Pi PC,
> but that will change to a different H3 platform in due course), and displaying
> on a 1080p/60 monitor connected via hdmi.
> 
> My current gstreamer pipeline is gst-launch-1.0 -v udpsrc port=5600 !
> "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-
> name=(string)H264, payload=(int)96" ! rtph264depay ! h264parse ! v4l2slh264dec
> ! kmssink plane-id=31 plane-properties=s,zpos=3
> 
> if I change to fakesink, the decode is working beautifully (and according to
> gstreamer latency stats is giving a decode time  of around 5ms - better than I
> had hoped!) However I am having issues with kmssink. Initially I was only
> getting 30fps display, but applying the kmssink fix for atomic drm clients
> from https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/issues/654 and
> also the rockchip kernel fix I get smooth 60fps decoding, but only if I add a
> ! queue ! element between  v4l2slh264dec and kmssink which pushes the latency
> up unacceptably high.
> 
> Playing with ! queue ! output buffers (max-size-buffers=1 leaky=downstream)
> shows it needs max-size-buffers=3 buffers before it displays smoothly. (at
> max-size-buffers=1 I alternately 1 frame dropped then 2 frames dropped, at
> max-size-buffers=2 it's one frame dropped then no frames dropped) 

I can only presume that render time correctness does not matter to you. So my
advise would be to use kmssink sync=0. The timestamp are not very accurate
without an rtpjitterbuffer, and rtpjitterbuffer cannot be very accurate if you
don't allow enough latency.

The queue allow the latency to grow, but it does not add latency. You should
keep the queue, this will work around the starvation problem you'd get, as
otherwise you are not decoding in the gap of time between the frame being ready
for output a vblank picking the new frame.

> 
> I presume this means kmssink is not sending back memory buffers fast enough to
> be reused?
> Can anyone set me on the right track to getting ultra-low latency decoding
> working? For this application I don't need to worry about screen tearing or
> similar artefacts, just getting this video on screen as soon as I possibly
> can.
>  
> Thank you for any help