Advice needed on plugin state transitions

[Sebastian Dröge]

On Sun, 2022-08-28 at 21:08 +0200, Philippe Rouquier via gstreamer-devel wrote:
> Hello,
> 
> I'm developping a plugin that does speech to text
> (https://github.com/PhilippeRo/gst-vosk).
> 
> Before performing voice recognition the plugin needs to load large
> language models (>1.6G). The plugin does this in a thread while
> transitioning from READY to PAUSED.
> When the pipeline is started (pulsesrc ! queue ! my STT plugin !
> fakesink), there is the request for state change from READY to PAUSED,
> the plugin returns GST_STATE_CHANGE_ASYNC, sends an ASYNC_START
> message, starts the thread, loads the language model and when the
> language model is loaded, sends an ASYNC_DONE message and calls
> gst_element_continue_state().
> 
> That works well. The queue element right before the STT plugin stores
> the buffers while the model is being loaded and, when ready, the STT
> plugin can perform recognition on everything that has been recorded
> since the pipeline started and "catch up".

That makes sense but keep in mind that async state changes can be a bit
fragile at time, especially in live pipelines.

It would seem equally fine to load everything synchronously in
NULL->READY, for example.

> Now, my problem is that if the STT plugin needs to load a new model,
> while the pipeline is playing, I don't really know what to do.
> 
> My first idea was to set the STT plugin back to READY internally -
> which cleans the current model during the transition from PAUSED to
> READY - and then call gst_element_sync_state_with_parent() which
> triggers a new call to set_state(PAUSED) that leads to the
> transitioning described above. Except it does not work since after
> loading the new model, sending the ASYNC_DONE message and calling
> gst_element_continue_state(), the whole pipeline seems inactive and the
> STT plugin's chain function is no longer called. Another "symptom" is
> that the sinkpad no longer has any caps set.

You can't really change the element state from inside an element,
that's not going to work.

You could make use of gst_element_lost_state() and API like that, but
getting that to behave right is not trivial. GstBaseSink is the only
user for that IIRC.

> Now, the other solution I'm using is to leave the STT plugin's state as
> is (playing) and just perform the loading of the model as described
> above (ASYNC_START message, thread, loading, ASYNC_DONE message,
> continue_state()).

See above, and look at GstBaseSink for an example of doing this.

I don't really have any other suggestion here but keep in mind that in
live pipelines this will not behave great. It might be better to just
do re-loading in the background (ideally in advance before you need
it), and continue working with the old models (if possible) until
everything new is loaded? Or to queue up input until that point.

-- 
Sebastian Dröge, Centricular Ltd · https://www.centricular.com