How to mix live and non-live sources

[Marianna Smidth Buschle]

Thanks for the help


I have set the offset on the src pads of the decoders (for both audio 
and video) and I got some progress but still have some issues.

 From what I can see it relies on the Segment event? I seem to see a new 
segment (with a base equal to the offset value) event when I set the pad 
offset.


So I think the problems I still have might be on the pipeline decoupling 
element (interpipesrc/sink) I'm using not handling the Segment correctly.

I have founds some problems in their code and corrected that, but might 
still be missing something.


Specifically my problem is like this: I'm listening to the live source 
but want to change to a replay sequence.

The replay sequence is a short transition video, then the replay stream 
itself, followed by the transition video again and then back to live.


A simplification of the topology:

v4l2src -> interpipesink name=vlive

pulsesrc -> interpipesink name=alive

filesrc location=transition.ts -> tsdemux -> h264_dec -> interpipesink 
name=vtransition

                                  ->  acc_dec -> interpipesink 
name=atransition

filesrc location=cam.ts -> tsdemux -> h264_dec -> interpipesink name=vreplay

                              ->  aac_dec -> interpipesink name=areplay

interpipesrc name=vselect listen-to=vlive -> h264_enc -> tsmux -> 
rtpmp2tpay -> udpsink

interpipesrc name=aselect listen-to=alive -> aac_enc  ->


So when I want to do a replay I start by preparing the replay file 
pipeline stream by playing, seeking and then pausing (because I found 
out that seeking takes time and I need seamless changes).

Then I play the transition file pipeline, set the pad offset on the 
decoders::src and set the interpipesrc to listen-to vtranstion/atransition.

When I get the EOS I play the replay file pipeline, set the pad offset 
on the decoders::src and set the interpipesrc to listen-to vreplay/areplay.

When I get next the EOS I set the transition file pipeline to NULL, then 
to PLAYING, set the pad offset on the decoders::src and set the 
interpipesrc to listen-to vtranstion/atransition.

On this final EOS I just set the interpipesrc to listen-to vlive/alive 
again.


And the problem is that I see a new segment when I play the transition 
file pipeline the 1st time and also when I play the replay file 
pipeline, but not when I play the transition file pipeline the 2nd time.

So the tsmux accepts the frames of the 1st transition and the replay 
itself, but complains about backwards DTS for the end transition 
(because there is no new Segment with a proper offset).


So, is there something wrong going on the interpipe, or do I need to 
"reset" the transition file pipeline on a different manner than setting 
to NULL in order to be able to get the new Segment?


  On 24.09.2021 13.23, Sebastian Dröge wrote:
> On Fri, 2021-09-24 at 13:17 +0200, Marianna Smidth Buschle wrote:
>> On 24.09.2021 13.05, Sebastian Dröge wrote:
>> On Fri, 2021-09-24 at 12:59 +0200, Marianna Smidth Buschle via 
>> gstreamer-devel wrote:
>>
>>> I have a use case where I have N live sources (v4l2src) from which I 
>>> can
>>> select to be the output.
>>>
>>> I also need to be able to make a replay (seek from a filesrc).
>>>
>>> My question is how to properly handle the mixing of the live and
>>> non-live sources since the timestamps don't match.
>>>
>>> The filesrc starts from zero (or the seeking point) and the live 
>>> sources
>>> have a running-time based on the clock.
>>
>> This is actually simpler than what you wrote. What you would do is to 
>> call gst_pad_set_offset() on the source pad of the non-live streams 
>> that you want to connect to the mixer.
>> That would then make sure to update the running time accordingly.
>>
>> Can I do that directly on the filesrc (the MPEG-TS) stream?
>>
>> Or should it be on the tsdemux pads, or later on the audio/video 
>> decoder pads?
>>
> filesrc doesn't know anything about time, so that won't have any effect.
> I would do it directly on the source pads that you connect to 
> compositor/audiomixer/etc.
>
> -- 
> Sebastian Dröge, Centricular Ltd · https://www.centricular.com 
> <http://www.centricular.com>
>
-- 
Best regards / Med venlig hilsen
“Marianna Smidth Buschle”

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20211001/fbb1b528/attachment-0001.htm>