Advice needed on plugin state transitions

[Philippe Rouquier]

Thanks a lot for your suggestions !

Le mardi 30 août 2022 à 16:46 +0300, Sebastian Dröge a écrit :
> On Sun, 2022-08-28 at 21:08 +0200, Philippe Rouquier via gstreamer-devel wrote:
> > 
> > That works well. The queue element right before the STT plugin stores
> > the buffers while the model is being loaded and, when ready, the STT
> > plugin can perform recognition on everything that has been recorded
> > since the pipeline started and "catch up".
> 
> That makes sense but keep in mind that async state changes can be a bit
> fragile at time, especially in live pipelines.
> 
> It would seem equally fine to load everything synchronously in
> NULL->READY, for example.

Am I right to suppose it would block the call to gst_element_set_state
(pipeline, READY/PAUSED/PLAYING) made by the application until the
model is fully loaded?
Or would the loading be done in a thread? Maybe the streaming thread
since if a queue element is used there is a new thread created on the
queue's srcpad which could block and still allow the pulsesrc connected
to the queue's sinkpad to continue recording.

> > Now, my problem is that if the STT plugin needs to load a new model,
> > while the pipeline is playing, I don't really know what to do.
> > 
> > My first idea was to set the STT plugin back to READY internally -
> > which cleans the current model during the transition from PAUSED to
> > READY - and then call gst_element_sync_state_with_parent() which
> > triggers a new call to set_state(PAUSED) that leads to the
> > transitioning described above. Except it does not work since after
> > loading the new model, sending the ASYNC_DONE message and calling
> > gst_element_continue_state(), the whole pipeline seems inactive and the
> > STT plugin's chain function is no longer called. Another "symptom" is
> > that the sinkpad no longer has any caps set.
> 
> You can't really change the element state from inside an element,
> that's not going to work.
> 
> You could make use of gst_element_lost_state() and API like that, but
> getting that to behave right is not trivial. GstBaseSink is the only
> user for that IIRC.
> 

I will try that and see how GstBaseSink handles it. Thanks for the
pointer.

> > Now, the other solution I'm using is to leave the STT plugin's state as
> > is (playing) and just perform the loading of the model as described
> > above (ASYNC_START message, thread, loading, ASYNC_DONE message,
> > continue_state()).
> 
> See above, and look at GstBaseSink for an example of doing this.
> 
> I don't really have any other suggestion here but keep in mind that in
> live pipelines this will not behave great. It might be better to just
> do re-loading in the background (ideally in advance before you need
> it), and continue working with the old models (if possible) until
> everything new is loaded? Or to queue up input until that point.

For the moment, that's what I have, an internal queue for the incoming
buffers that are then consumed when model/recognizer are ready.