Seeing huge volume of buffers with HLS/TS source compared to RTSP sources

[Matt Feury]

Looks like having `identity sync=true` directly after the source element
fixes things really nicely here! I'm now seeing buffers per second match
the FPS value with HLS streams. This line in the docs seems to be most
relevant:

> Synchronization is now a matter of making sure that a buffer with a
certain running-time is played when the clock reaches the same running-time

Not totally an expert here in Gst's clock, so would welcome feedback if
folks think this is a "Bad Idea", but right now it seems to be working
really well.

Thanks for all the help here.

Matt

On Wed, Jan 18, 2023 at 9:15 PM Matt Feury <mattfeury at gmail.com> wrote:

> Hey Tim, thanks for the context and great questions.
>
> I had a similar thought. RTSP streams itself "live" whereas the HLS stream
> is served using .m3u8 files which are playlists of ts files. So I do think
> perhaps the souphttpsrc is internally fetching the ts files as they are
> added to the playlist in realtime, but those files are likely processed
> "all at once" since it's then just a download of a .ts file. I'm not sure
> how this could lead to what I'm seeing, but it is a good direction I should
> keep chasing down. To answer your question, the HLS stream is a
> "live" stream as well (it's supposed to be real time and i have even set
> is-live on the souphttpsrc), but to my knowledge that's still served in a
> playlist of .ts fragments, so will naturally be more latent.
>
> I put an identity directly after the source (either souphttpsrc or
> rtspsrc), and I am seeing buffers of size 500-1000 (roughly) with
> souphttpsrc, whereas rtspsrc buffers are of size 50,000-100,000. so it
> "makes sense" that the same h264 data would require *more *buffers if
> they're smaller (hls) but less if they're bigger. But I just need to figure
> out how to "combine them" so to speak.
>
> I am going to play with clocksync and other sync attributes as well.
>
> Feel free to correct any of my misunderstandings above. I'm fairly new to
> the gst world but really enjoying digging in :).
>
> thanks
> Matt
>
>
> On Wed, Jan 18, 2023 at 7:06 PM Tim-Philipp Müller <t.i.m at zen.co.uk>
> wrote:
>
>> On Wed, 2023-01-18 at 17:30 -0500, Matt Feury via gstreamer-devel wrote:
>>
>> Hi Matt,
>>
>> I'm currently running a pipeline that can support either rtspsrc or HLS
>> sources (via souphttpsrc). The pipeline generally looks something like this:
>>
>> > rtspsrc/souphttpsrc (depending on url structure) ! parsebin ! decodebin
>> ! videoconvert ! identity ! tee ! queue ! multiple sinks (e.g. appsink,
>> webrtc, etc, all from tee).
>>
>> Note this has been scrubbed of detailed pipeline info like specific sinks
>> and extra queues/identities for simplicity's sake.
>>
>> I am using the identity element to calculate the number of buffers per
>> second passing through. When running the pipeline with an RTSPsrc, i am
>> seeing a BPS value that corresponds heavily to the FPS (e.g. 5). However,
>> when I run with an HLS source, i am seeing BPS values 10-100x as large as
>> FPS (e.g. 250).
>>
>> Weirdly, in my appsink, if i just do a simple cv2.imwrite to the local
>> machine, I can confirm that each buffer coming through corresponds to a
>> unique frame of the stream.
>>
>> So I guess my question is: what could be causing these increased buffer
>> counts for the same pipeline but only when HLS/TS data is flowing through
>> it? While the stream sources are different, the FPS of each is roughly
>> similar. I would presume that after the decodebin and videoconvert, I am
>> left with raw video of the same general media type / caps regardless of the
>> input source, so I am surprised to see such variation.
>>
>>
>> Only thing I can think of is that rtspsrc is live and naturally receives
>> an input stream at the playback rate, whereas with your hls setup maybe
>> you're downloading fragments as fast as your bandwidth allows?
>>
>> is your HLS stream a live stream or a VOD stream? Do you get the higher
>> fps only at the beginning or continuously over a longer period?
>>
>> For non-live inputs synchronisation only happens at the sink(s)
>> typically, and it will depend on the sink in use whether it's live or not
>> by default (sync=true on the sink).
>>
>> For appsink you might alaso want to limit the internal queue which by
>> default is unlimited.
>>
>> You can play with identity sync=true (or replace with a clocksync
>> element) to see if that makes a difference (I don't think that should be
>> necessary though).
>>
>> Cheers
>>  Tim
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20230119/67180566/attachment.htm>