GStreamer - pipeline design - multi-threading

Fri Jun 2 10:25:44 UTC 2017

On Tue, May 30, 2017 at 4:25 PM, Nicolas Dufresne <nicolas at ndufresne.ca>
wrote:

> Le mardi 30 mai 2017 à 11:02 +0200, Bogdan Pavkovic a écrit :
> > Hi all,
> >
> > I realized that an attachment cannot be seen from the archives so here
> is the link just in case...
> >
> > https://mega.nz/#!h8wFVarA!leTOvjLg4VhBxX-wNPY3_w8jcFE-W-vspXw7FOT_SAU
> >
> > Best,
> > B
> >
> > > On Tue, May 30, 2017 at 10:49 AM, Bogdan Pavkovic <bogprs at gmail.com>
> wrote:
> > > Dear all,
> > >
> > > Here is the diagram for your convenience.
> > > I hope there is someone that can answer at least some subset of my
> questions.
> > >
> > > Thanks again in advance.
> > >
> > > Best,
> > > Bogdan
> > >
> > >
> > > > > On Fri, May 26, 2017 at 1:15 PM, Bogdan Pavkovic <bogprs at gmail.com>
> wrote:
> > > > Dear all,
> > > >
> > > > I am a newby to GStremer Community
> > > > but I managed to thoroughly go through ALL offered tutorials.
> > > >
> > > > GOAL: I need to design a performance-critical pipeline and create a
> couple of new filter plugins.
> > > > Any help will be appreciated.
> > > >
> > > > General pipeline is divided in 3 bins: input, core, and output.
> > > > For the input part we have already completed 4 options: RTP client
> (existing plugin), MJPEG, V2L, and FILE (existing).
> > > > For the output part we use Wayland.
> > > >
> > > > Core part is processing which roughly is done in 3 stages:
> > > > pre-processing, heavy processing, and some sort of Kalman filtering.
> > > > Q1: My idea is to make 3 plugins that will form a larger bin. Is
> this the right way to go?
> > > > Pre-processing plug-in need to have on-request sink pads since
> number of input video streams can vary between 1 and 4.
> > > > Source (output) pads (on request) will be created and linked to
> consecutive elements on the fly as they appear.
>

Bonjour Nicolas. Merci pour ta réponse.
Find my answers below.

>
> Is there a relationship between the input streams ? Since having a
> single element in the context were there is not link between each
> input/output pair will just make your pipeline more complex to handle.
> You will endup having N+1 thread making call into your element. You
> could though use a bin just to group and manage the internal filters.
>

Input streams are not in any relationship. Each comes from separate camera
and after same processing stages (3 in total as described) will be
displayed in 2 different windows.

If I understand correctly your answer, for each PAD in an element, there
will be a separate thread called to handle the *_chain *function every time
a buffer arrives.

>
> > > > Q2: Do all consecutive elements in Core bin need to have on request
> pads due to non-deterministic number of input videos?
>
> If I understand correctly, you are asking if the element inside the bin
> need to have request pads. If that is correct, the answer is no. You
> can have a bin with request sink pads, that will use ghost pad to
> interface with internal filters static sink pads. The bin can then
> notify sometimes src pads by ghosting a filter static src pad.
>

Understood. We will then implement 3 filters with N static sink pads and
map them with ghost pads to a bin.
What about src pads of an element inside bin? Do they need to be created on
request or as a simple static pads?

>
> > > >
> > > > Heavy processing plug-in needs to do heavy lifting and we intend to
> use multi-threading on 4 available cores.
> > > > Q2: Do we need to use 2 queue elements before each of the pads of
> this element to separate processing of each video stream?
>
> It depends of how you implement the parallel processing. If you can do
> parallel processing on a single GstBuffer, then you don't need any
> queuing. This method is usually used to achieve low latency processing
> as queuing multiple buffers to feed multiple threads will introduce
> latency.
>

Clear. Since we will do parallel processing on a single GstBuffer then we
will not use a queue element.

>
> > > > Q3: I understood there are tasks that can do further
> multi-threading. Can we use them behind a queue element?
>
> I'm not sure I understand this question. What do you mean by tasks ?
> You can of course create and manage threads in your element.
>

I meant GTask but you've already explained below.

>
> > > > Q4: If we use 4 tasks, is there a way to signal and sync all of
> them? i.e. Do we get a signal  when all 4 task outputs are ready?
>
> I'm not sure this question correlate with anything in GStreamer.
> GstTask is a wrapper around a thread, with proper state that let you
> start/stop/pause the task and link it to a thread pool. Signalling that
> data is ready has to be done by your code, could be using GCond or some
> advance queue data structure.
>

Understood. GCond seems like an elegant way.

>
> > > > Q5: How can we configure a task to target a specific core?
>
> This is doable in a portable way, hence not expose through the GThread
> abstraction. In general, you should trust your scheduler not to move
> the thread around in a way that it would degrade the performance. If
> you are implementing this only for unix system, you can of course
> implement your own, and use pthread API directly (see
> pthread_setaffinity_np).
>

Great! Looks exactly like what we need.

>
> > > >
> > > > Finaly, Kalman plugin is not calculation intensive and can handle
> both streams.
> > > > Q6: Can we somehow undo the queue element? Does it make sense?
>
> What does "undoing" an element means ? If by chance you are referring
> to the requires queues after demuxer elements, be aware that this is
> needed because demuxers receives buffer from a single streaming thread
> and does not create any new threads for the output. The queues are
> needed so that each branch of the demuxer can work independently. If
> you have input/output pairs in your bins, you'll have 1 streaming
> thread per pair, hence each output will already work independently from
> each other.
>

Actually, this question became redundant with your previous answers.
Thanks for giving details about demuxers.

Best,
Bogdan

>
> > > >
> > > > I can provide a diagram of the proposed pipeline if needed to
> clarify things.
> > > >
> > > > Many thanks in advance.
> > > >
> > > > Best regards,
> > > > Bogdan
> > > >
> > >
> > >
> >
> > _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20170602/35152493/attachment-0001.html>