4K playback optimization help needed

Mon Jan 27 08:36:04 UTC 2020

Thanks everyone for your helpful replies.

I've tried most of what you've suggested, and while in the end the situation
hasn't improved, I might have some new information. Most importantly:

1) Without "queue" elements, some frames were being dropped due to QoS,
however, with queues included, I didn't see any "dropping frame" warnings.

Pipeline A without queues: "gst-launch-1.0.exe filesrc location=4K.mkv !
matroskademux ! h265parse ! nvh265dec ! glimagesink --gst-debug 2 2>dbg.txt"

Pipeline B with queues: "gst-launch-1.0.exe filesrc location=4K.mkv !
matroskademux ! queue max-size-bytes=0 ! h265parse ! queue max-size-bytes=0
! nvh265dec ! queue max-size-bytes=0 ! videorate ! video/x-raw(ANY),
framerate=(fraction)60/1 ! glimagesink sync=false --gst-debug 2 2>dbgq.txt"

Also, separately, I created debug logs with "video*:6", "glimagesink:6",
etc. for each involved element. Separately, as to keep log files specific to
individual components, and to avoid slowing things down too much
(--gst-debug 6 is way too slow).

a) The log file created by video*:6 had some dropped frame warnings for
pipeline A, not for B:

"0:00:01.407290000 12224 00000225466CC040 WARN            videodecoder
gstvideodecoder.c:3207:gst_video_decoder_clip_and_push_buf:<nvh265dec0>
Dropping frame due to QoS. start:0:00:00.333666664
deadline:0:00:00.333666664 earliest_time:0:00:00.360051535"

b) In addition, log files for "--gst-debug 2", both A and B, contained one
type of warning many times (copied 4 here to show time progression). This is
probably not the reason for the "jumpiness", see CPU decoding in #2.

"0:00:00.890107000  5644 000002C61149C040 WARN      matroskareadcommon
matroska-read-common.c:712:gst_matroska_read_common_parse_skip:<matroskademux0:sink>
Unknown CueTrackPositions subelement 0xf0 - ignoring
0:00:00.890587000  5644 000002C61149C040 WARN      matroskareadcommon
matroska-read-common.c:712:gst_matroska_read_common_parse_skip:<matroskademux0:sink>
Unknown CueTrackPositions subelement 0xf0 - ignoring
0:00:00.891072000  5644 000002C61149C040 WARN      matroskareadcommon
matroska-read-common.c:712:gst_matroska_read_common_parse_skip:<matroskademux0:sink>
Unknown CueTrackPositions subelement 0xf0 - ignoring
0:00:00.891540000  5644 000002C61149C040 WARN      matroskareadcommon
matroska-read-common.c:712:gst_matroska_read_common_parse_skip:<matroskademux0:sink>
Unknown CueTrackPositions subelement 0xf0 - ignoring"

No other warnings were present in any of the other log files. I suspect that
perhaps frame dropping isn't the issue here. Playback is still "jumpy", more
so than I get with MPC-BE. Tbh, I can see small steps from frame to frame in
both gstreamer and MPC-BE output, however, gstreamer playback seems to show
some bigger "jumps" all over the screen a few times a second, and MPC-BE
gives a smoother feeling overall.
Is it possible that the larger "jumps" are caused by glimagesink being
"slow", or lacking vsync? I'd love to try out another image sink, but don't
know of any that could show 

2) CPU decoding is smooth

a) "gst-launch-1.0.exe filesrc location=4K.mkv ! matroskademux ! h265parse !
avdec_h265 ! d3d11videosink"

On my (strong enough) work PC, this pipeline works better, smoother than the
one above. It's just as smooth as MPC-BE, even without queues. BtW,
matroskademuxer has the same warnings for the CPU pipeline as above, the
playback is smooth nonetheless.

b) Uploading the CPU-decoded video to the GPU and displaying with
glimagesink has the same problems as above:

"gst-launch-1.0.exe filesrc location=4K.mkv ! matroskademux ! h265parse !
queue max-size-bytes=0 ! avdec_h265 ! videoconvert ! queue max-size-bytes=0
! glupload ! queue max-size-bytes=0 ! glimagesink"

c) I've also tried to do the other way, as in GPU decoding first and
replacing glimagesink with another video sink, but can't find any way to use
another video sink. E.g.:

"gst-launch-1.0.exe filesrc location=4K.mkv ! matroskademux ! h265parse !
nvh265dec ! gldownload ! videoconvert ! autovideosink" only shows a green
image. "d3d11videosink" is the same (perhaps autovideosink used that? didn't
check).

All in all, I suspect the problem is not with frames being dropped, but
somehow the display of glimagesink is "jumpy". However, glimagesink isn't
limited by performance either, because with "sync=false" it plays back
frames very quickly and with no apparent image quality issues (jumps small
or large). BtW, the "max-lateness=0", "qos=false" parameters don't help
glimagesink playback at all, if they're even relevant.

Can you suggest please:
- How to continue exploring the possible issues?
- Is there any other video sink I could try instead of glimagesink to see if
that's the problem? Alternatively, instruct  me how to make
d3d11videosink/autovideosink work after gldownload.

**********************
Let me quickly also respond to a few other things you mentioned:

3) "lateness"

@Matteo wrote:
> You should check the latency of your pipeline. It is possibile that the
> computed latency is on the edge of the rendering deadline.

Based on my, admittedly not very deep, knowledge of gstreamer, latency is
relevant to live streams, whereas (according to gstreamer docs) "For
pipelines where the only elements that synchronize against the clock are the
sinks, the latency is always 0". Was perhaps "max-lateness", what you meant?
In any case, I feel your suggestion is similar to what's been discussed in
the thread I referred to in my original post, i.e. that frames might be
dropped because some processing step takes too long (in my understanding). I
was trying to get around that by setting "sync=false" and controlling the
frame rate with the "videorate" element - not sure if it has the correct
effect, but it improved the playback quality a bit. I've also tried to apply
"max-lateness=-1" to the glimagesink, with no visible effect. That might be
because  I suspect frames aren't really being dropped(?).

4) glimagesink vsync

@Nicolas wrote:
>Note that there is no "vsync" property on glimagesink in mainline
GStreamer. Please review all your applied patches as they might also be the
cause of all your troubles.

I'm using gstreamer compiled with Cerbero master (1.17) with visualstudio,
nvcodec and intelmsdk variants enabled. No special patches I know of.
Glimagesink might not have a vsync parameter, but it's not complaining about
it either when I use it. Should it? "fullscreen=true" doesn't seem to work
with it either but it's not showing any error message when it's added.

5) recent merge removing nvcodec latency

@Nicolas wrote:
> https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/merge_requests/998
> It's not much, but it removes 1 frame latency on the decoder.

I've re-compiled gstreamer with Cerbero/master today, hopefully that
included this patch. Otherwise I'm not sure how to apply it.

--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/