How to reliably get samples from appsink before PTS

[Tamas]

Hi everyone,

I'm trying to get frames from a pipeline's appsink, do some
post-processing, and then draw the frames somewhere. I decided to present
the frames at or nearest their PTS according to the appsink's clock.
Post-processing takes some time, and I really don't want any unnecessary
delay between PTS and presentation, so I'd prefer to receive the frames
earlier than later.

However, the timings of when the callback attached to the appsink's
"new-sample" signal is called seem to vary, sometimes before the frame's
PTS, and in some cases after. If it were a sink that presents the frames on
a display, I'd say that perhaps the timings follow the screen's refresh or
otherwise related to when the frames can actually be presented, but an
appsink doesn't have such limitations, so why the erratic behaviour,
especially why the delay? I'd like to have some control over this behaviour
- tell the appsink to provide the frames earlier, but I'm not aware of such
a way.

To get more insight, I also attached a probe to the appsink's "sink" pad.
What happens, in general, is that the appsink receives the next buffer soon
after one was pulled from it (by the new-sample callback), and then it
waits some time before emitting "new-signal". In some media, it usually
happens before the PTS, in others it can be later.

For example, I got this data from playing a 4K H264 movie:

(Pipeline: uridecodebin3 ! videoconvert ! glupload ! glcolorconvert !
glcolorscale ! queue ! appsink)

<PROBE      #  7>   PTS=291000000   tsink=224458200 sink pos=224461800
<NEW SAMPLE #  7>   PTS=291000000   tsink=255952100 sink pos=255954200

<PROBE      #  8>   PTS=333000000   tsink=256490400 sink pos=256491900
<NEW SAMPLE #  8>   PTS=333000000   tsink=303961700 sink pos=303965300

<PROBE      #  9>   PTS=375000000   tsink=304605500 sink pos=304608300
<NEW SAMPLE #  9>   PTS=375000000   tsink=335957100 sink pos=335961800

<PROBE      # 10>   PTS=417000000   tsink=336501600 sink pos=336503700
<NEW SAMPLE # 10>   PTS=417000000   tsink=383959700 sink pos=383961800

<PROBE      # 11>   PTS=458000000   tsink=384398800 sink pos=384400400
<NEW SAMPLE # 11>   PTS=458000000   tsink=431950300 sink pos=431954700

<PROBE      # 12>   PTS=500000000   tsink=432523800 sink pos=432527200
<NEW SAMPLE # 12>   PTS=500000000   tsink=463954000 sink pos=463956900

<PROBE      # 13>   PTS=542000000   tsink=464436300 sink pos=464438800
<NEW SAMPLE # 13>   PTS=542000000   tsink=511979400 sink pos=511984000

<PROBE      # 14>   PTS=583000000   tsink=512579000 sink pos=512581400
<NEW SAMPLE # 14>   PTS=583000000   tsink=543973700 sink pos=543976100

tsink := gst_element_get_current_running_time( appsink )
sink pos := gst_element_query_position( appsink, GST_FORMAT_TIME, &sinkpos
);
Here, new-sample signals are sent some time before PTS.

Another example, MJPG movie @ 2 fps:

<PROBE      #  2>   PTS= 500000000  tsink=    777500 sink pos=    780400
<NEW SAMPLE #  2>   PTS= 500000000  tsink= 512079000 sink pos= 512081600

<PROBE      #  3>   PTS=1000000000  tsink= 512563100 sink pos= 512565400
<NEW SAMPLE #  3>   PTS=1000000000  tsink=1007110000 sink pos=1007112200

<PROBE      #  4>   PTS=1500000000  tsink=1007577100 sink pos=1007579800
<NEW SAMPLE #  4>   PTS=1500000000  tsink=1502023500 sink pos=1502025500

<PROBE      #  5>   PTS=2000000000  tsink=1502778200 sink pos=1502780700
<NEW SAMPLE #  5>   PTS=2000000000  tsink=2013087500 sink pos=2013089600

<PROBE      #  6>   PTS=2500000000  tsink=2013675000 sink pos=2013678200
<NEW SAMPLE #  6>   PTS=2500000000  tsink=2508031100 sink pos=2508033300

<PROBE      #  7>   PTS=3000000000  tsink=2508494900 sink pos=2508496200
<NEW SAMPLE #  7>   PTS=3000000000  tsink=3001936200 sink pos=3001939100

Here, for some reason, the appsink waited until AFTER the PTS to send the
signal. What causes appsink to delay nearly all frames?

Actually, the why isn't so important as how I could get a more consistent
behaviour from appsink.

Right now I have several possible solutions (below), but I'm writing to ask
what would be a good solution?

I could...

1. Pull samples from the "new-sample" callback; don't try to present at PTS
but accept a delay (e.g. 1 frame delay); but how do I know how much is safe
to wait? And I don't like having a delay at the start of playback for
several reasons.

2. "Steal" buffers from the sink pad to receive them much earlier - this is
what I'm doing now, but would it not break when 1 buffer != 1 sample?

3. Modify PTS to something earlier at the sink pad, then hope that appsink
would emit new-sample earlier; present them with correct PTS. Again,
appsink is not consistent so I'm not sure what to expect from this.

4. sync=false on the appsink might help but then I'd be responsible for all
aspects of timing; I rather like that the appsink takes care of a lot of
that for me and calls back for new samples

Cheers
Suggestions are welcome.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20230129/11719a9f/attachment.htm>