[gst-devel] Video processing
Aurelien Grimaud
gstelzz at yahoo.fr
Sat Aug 30 00:10:56 CEST 2008
Hi,
Queue starts to leak when full
If you want one buffer at a time, try and make your queue 1 buffer long.
What about this ?
gst-launch -v videotestsrc is-live=TRUE ! queue max-size-buffers=1
leaky=2 ! identity sleep-time=1000000 ! fakesink
identity sleeps 1 sec, simulating 1 sec video processing.
Queue leaks downstream (old buffers) and is 1 buffer long, so it keeps
only the last one.
The problem might come from the wrong timestamps received by the imagesink.
There will be gaps, I do not know how it will handle it.
Aurelien
Bruno a écrit :
> Okay thanks a lot for these precious informations.
>
> With a pad probe I was able to draw on the xvimagesink buffer, so that
> part is okay now.
>
> But I still can't find how to drop the frame sent by camera during the
> image processing,
> I found that the queue element has a property called "leaky" which,
> seeing the description of the element :
>
> "Where the queue leaks, if at all.
>
> Default value: Not Leaky"
>
> It should be what I'm looking for. If I change its value to 1, the queue
> element should drop the coming frames until the processing of the next
> element is not finished, isn't it ?
>
> I tried to change the value and didn't see any difference, so maybe I'm
> wrong.
>
> Thanks again.
>
> Bruno
>
>
>
>
>
>
> 2008/8/28, Eric Zhang <nicolas.m.zhang at gmail.com
> <mailto:nicolas.m.zhang at gmail.com>>:
>
> Hi, Bruno:
>
> I think that what you mentioned is not a gstreamer related
> topic. What you are worrying about is the performance of image
> processing and there is a lot of ways to improve it, such as
> re-consider the algorithm of image processing, try to parallel the
> processing and so on.
>
> The `queue' element in gstreamer is just a container of buffers.
> It just receive the buffer and push it out until `limitation'
> reaches(max-bytes, max-time, max-buffers). We usually use this
> element to cache the buffers and implement `buffering' features
> which is useful in a media player.
>
> Eric Zhang
>
>
> 2008/8/27 Bruno <botte.pub at gmail.com <mailto:botte.pub at gmail.com>>
>
> Ok thanks a lot Eric, I didn't know the buffer can be modified
> on the fly this way.
>
> But what I don't really understand is how exactly the pipeline
> works. If I do that way, will the pipeline take one frame from
> the camera, call the buffer probe related function, doing the
> image processing calculation, modify the buffer, then send it to
> the screen ? Because that processing takes a long time, and lot
> of frame are coming from the cam during the calculation. I'd
> like them to be dropped, then when the calculation from the
> first frame is finished, the image processing callback has to
> take the next coming frame. (or the one just before, but not the
> second one).
>
> Actually it seems that all frame are showed, like it does the
> calculation for the first, then keep the next coming in memory,
> and when the first is processed, it comes to the second. (or at
> least it seems like this when I start the prog). Maybe it's due
> to the "queue" element ? (I took this from the maemo-camera
> example, and I don't get what the queue element is for)
>
> Thanks again for explanations !
>
> Kind regards,
> Bruno
>
>
> 2008/8/27, Eric Zhang <nicolas.m.zhang at gmail.com
> <mailto:nicolas.m.zhang at gmail.com>>:
>
> Hi, gstreamer-devel:
>
> Hello, Bruno, actually I have not many experiences on
> this topic and I was just going to give you some clews. OK,
> AFAIK, you can add a buffer probe before the xvimagesink and
> check out the GstBuffer which is going to flow into it and
> add your rectangle on this GstBuffer, that's it. You don't
> need to draw the buffer yourself because xvimagesink will do
> it for you(just give your drawing area's window id to it,
> which you have achieved before).
>
> You can refer to Chapter 18 of gstreamer application
> develop manual, there is an example which inverts the image
> by adding a buffer probe and modify the buffer on the fly.
> Maybe this can help you a little.
>
>
> Eric Zhang
>
> 2008/8/26 Bruno <botte.pub at gmail.com
> <mailto:botte.pub at gmail.com>>
>
> Could you be a little more precise please ?
> How do I modify the buffer ? How do I draw it on the
> screen after ? should I still declare my screen widget
> from appdata struct as a drawing area ?
>
> Thanks for help
>
>
> 2008/8/26, Eric Zhang <nicolas.m.zhang at gmail.com
> <mailto:nicolas.m.zhang at gmail.com>>:
>
> Hi, gstreamer-devel:
>
> Oh, I think you can use a buffer/data probe to
> achieve this. Modify the buffer, add a rectangle on
> every frame. Or, modify the xvimagesink. :)
>
> Eric Zhang
>
> 2008/8/26 Bruno <botte.pub at gmail.com
> <mailto:botte.pub at gmail.com>>
>
> Yep that is what I was doing before. Do you
> think I can draw rectangles over the cam image
> when using xvimagesink ?
>
>
> 2008/8/26, Eric Zhang <nicolas.m.zhang at gmail.com
> <mailto:nicolas.m.zhang at gmail.com>>:
>
> Hi, gstreamer-devel:
>
> If you only want to `ximagesink' or
> `xvimagesink' draws images in
> your GtkDrawingArea, there is a very simple
> way to achieve this:
>
> Just connect the `expose-event' signal
> of GtkDrawingArea and pass
> the window ID to the sink element:
>
> // Drawing on our drawing area
> g_signal_connect(G_OBJECT(area),
> "expose-event", G_CALLBACK(expose_cb),
> NULL);
>
> /* Callback to be called when the drawing
> area is exposed */
>
> static gboolean expose_cb(GtkWidget *
> widget, GdkEventExpose * event,
> gpointer data)
> {
>
> // `play-videosink' is your video sink
> element
> gst_x_overlay_set_xwindow_id(GST_X_OVERLAY(play->videosink),
>
>
> GDK_WINDOW_XWINDOW(widget->window));
> return FALSE;
> }
>
>
> That's it. If you want to draw the image
> yourself while not using
> `xvimagesink' or `ximagesink', then I think
> this is a Gtk+ problem, not
> a gstreamer issue.
>
>
> Eric Zhang
>
>
>
>
> Bruno wrote:
> > Hello all,
> >
> > I still have some questions about gstreamer.
> >
> > Actually I'd like to change the way my
> program work, in order to
> > display a frame from the camera only once
> the image processing has
> > been done, and with a rectangle over the
> face of the person.
> >
> > So I changed my pipeline (removed the
> screen_sink element), and I'd
> > like to send a buffer from my image
> processing function to the gtk
> > drawing area where the camera image was
> displayed before. I tried to
> > do it with gtk drawing area (and with gtk
> image too with no sucess),
> > but I can't find the way to change the
> image contained in the drawing
> > area.
> >
> > Here is my code :
> >
> >
> >
> > ///// IMAGE PROCESSING CALLBACK
> >
> > /* Callback to be called when data goes
> through the pad */
> > static gboolean process_frame(GstElement
> *video_sink,
> > GstBuffer *buffer, GstPad *pad,
> AppData *appdata)
> > {
> > int x, y;
> > // getting the pointer to camera buffer
> > unsigned char *data_photo =
> (unsigned char *)
> > GST_BUFFER_DATA(buffer);
> >
> >
> > // REMOVED PART WHERE THE COORDINATES OF
> THE POSITION OF THE FACE IS
> > CALCULATED //
> >
> >
> > // THIS PART IS WHAT I TRIED, BUT I HAVE A
> SEGMENTATION FAULT WHEN
> > CREATING PIXBUF //
> > GdkPixbuf *newscreen;
> > //newscreen =
> gdk_pixbuf_new_from_data(data_photo,
> > //GDK_COLORSPACE_RGB,
> /* RGB-colorspace */
> > //FALSE, /* No
> alpha-channel */
> > //8, /* Bits per
> RGB-component */
> > //IMAGE_WIDTH,
> IMAGE_HEIGHT, /* Dimensions */
> > //3*IMAGE_WIDTH, /*
> Number of bytes between lines
> > (ie stride) */
> > //NULL, NULL); /*
> Callbacks */
> >
> >
> > gdk_draw_pixmap(GDK_DRAWABLE(appdata->screen),
> > appdata->screen->style->black_gc,
> GDK_DRAWABLE(newscreen), 0, 0, 0, 0,
> > -1, -1);
> >
> > return TRUE;
> > }
> >
> >
> >
> >
> >
> > /////// PIPELINE
> >
> >
> > /* Initialize the the Gstreamer pipeline.
> Below is a diagram
> > * of the pipeline that will be created:
> > *
> > * |Camera| |CSP | |Screen| |Screen|
> |Image |
> > * |src |->|Filter|->|queue |->|sink |->
> |processing|-> Display
> > */
> > static gboolean
> initialize_pipeline(AppData *appdata,
> > int *argc, char ***argv)
> > {
> > GstElement *pipeline, *camera_src,
> *screen_sink;
> > GstElement *screen_queue;
> > GstElement *csp_filter;
> > GstCaps *caps;
> > GstBus *bus;
> > GstPad *sinkpad;
> >
> > /* Initialize Gstreamer */
> > gst_init(argc, argv);
> >
> > /* Create pipeline and attach a
> callback to it's
> > * message bus */
> > pipeline = gst_pipeline_new("test-camera");
> >
> > bus =
> gst_pipeline_get_bus(GST_PIPELINE(pipeline));
> > gst_bus_add_watch(bus,
> (GstBusFunc)bus_callback, appdata);
> > gst_object_unref(GST_OBJECT(bus));
> >
> > /* Save pipeline to the AppData
> structure */
> > appdata->pipeline = pipeline;
> >
> > /* Create elements */
> > /* Camera video stream comes from a
> Video4Linux driver */
> > camera_src =
> gst_element_factory_make(VIDEO_SRC,
> "camera_src");
> > /* Colorspace filter is needed to make
> sure that sinks understands
> > * the stream coming from the camera */
> > csp_filter =
> gst_element_factory_make("ffmpegcolorspace",
> > "csp_filter");
> > /* Queue creates new thread for the
> stream */
> > screen_queue =
> gst_element_factory_make("queue",
> "screen_queue");
> > /* Sink that shows the image on screen.
> Xephyr doesn't support XVideo
> > * extension, so it needs to use
> ximagesink, but the device uses
> > * xvimagesink */
> > //screen_sink =
> gst_element_factory_make(VIDEO_SINK,
> "screen_sink");
> >
> > sinkpad =
> gst_element_get_static_pad(screen_queue,"sink");
> >
> gst_pad_add_buffer_probe(sinkpad,G_CALLBACK(process_frame),
> appdata);
> >
> >
> > /* Check that elements are correctly
> initialized */
> > if(!(pipeline && camera_src /*&&
> screen_sink*/ && csp_filter &&
> > screen_queue))
> > {
> > g_critical("Couldn't create
> pipeline elements");
> > return FALSE;
> > }
> >
> >
> > /* Add elements to the pipeline. This
> has to be done prior to
> > * linking them */
> > gst_bin_add_many(GST_BIN(pipeline),
> camera_src, csp_filter,
> > screen_queue, /*screen_sink,*/
> NULL);
> >
> > /* Specify what kind of video is wanted
> from the camera */
> > caps =
> gst_caps_new_simple("video/x-raw-rgb",
> > "width", G_TYPE_INT, IMAGE_WIDTH,
> > "height", G_TYPE_INT, IMAGE_HEIGHT,
> > "framerate", GST_TYPE_FRACTION,
> FRAMERATE, 1,
> > NULL);
> >
> >
> > /* Link the camera source and
> colorspace filter using capabilities
> > * specified */
> >
> if(!gst_element_link_filtered(camera_src,
> csp_filter, caps))
> > {
> > return FALSE;
> > }
> > gst_caps_unref(caps);
> >
> > /* Connect Colorspace Filter -> Screen
> Queue -> Screen Sink
> > * This finalizes the initialization of
> the screen-part of the
> > pipeline */
> > if(!gst_element_link_many(csp_filter,
> screen_queue, /*screen_sink,
> > */NULL))
> > {
> > return FALSE;
> > }
> >
> > gst_element_set_state(pipeline,
> GST_STATE_PAUSED);
> >
> > return TRUE;
> > }
> >
> >
> >
> >
> >
> >
> >
> > /////// MAIN FUNCTION
> >
> >
> > int main(int argc, char **argv)
> > {
> > // variables for face detection
> > // main structure for vjdetect
> >
> > pdata = (mainstruct*) calloc(1,
> sizeof(mainstruct));
> > // Allocate memory for array of face
> detections returned by
> > facedetector (VjDetect).
> > pdata->pFaceDetections = (FLY_Rect
> > *)calloc(MAX_NUMBER_OF_FACE_DETECTIONS,
> sizeof(FLY_Rect));
> > init(pdata);
> >
> > AppData appdata;
> > appdata.expression = 0;
> > GtkWidget *hbox, *vbox_button, *vbox,
> *button1, *button2;
> >
> >
> > /* Initialize and create the GUI */
> >
> > example_gui_initialize(
> > &appdata.program, &appdata.window,
> > &argc, &argv, "Expression Detector");
> >
> > vbox = gtk_vbox_new(FALSE, 0);
> > hbox = gtk_hbox_new(FALSE, 0);
> > vbox_button = gtk_vbox_new(FALSE, 0);
> >
> > gtk_box_pack_start(GTK_BOX(hbox), vbox,
> FALSE, FALSE, 0);
> > gtk_box_pack_start(GTK_BOX(hbox),
> vbox_button, FALSE, FALSE, 0);
> >
> > appdata.screen = gtk_drawing_area_new();
> >
> gtk_widget_set_size_request(appdata.screen,
> 500, 380);
> > gtk_box_pack_start(GTK_BOX(vbox),
> appdata.screen, FALSE, FALSE, 0);
> >
> > button1 =
> gtk_toggle_button_new_with_label("Run/Stop");
> > gtk_widget_set_size_request(button1,
> 170, 75);
> >
> gtk_box_pack_start(GTK_BOX(vbox_button),
> button1, FALSE, FALSE, 0);
> >
> > button2 =
> gtk_toggle_button_new_with_label("Expressions
> ON/OFF");
> > gtk_widget_set_size_request(button2,
> 170, 75);
> >
> gtk_box_pack_start(GTK_BOX(vbox_button),
> button2, FALSE, FALSE, 0);
> >
> >
> > appdata.anger =
> gtk_image_new_from_file("./smileys/anger.jpg");
> >
> gtk_widget_set_size_request(appdata.anger,
> 160, 180);
> > appdata.disgust =
> gtk_image_new_from_file("./smileys/disgust.jpg");
> >
> gtk_widget_set_size_request(appdata.disgust,
> 160, 180);
> > appdata.fear =
> gtk_image_new_from_file("./smileys/fear.jpg");
> >
> gtk_widget_set_size_request(appdata.fear,
> 160, 180);
> > appdata.happy =
> gtk_image_new_from_file("./smileys/happy.jpg");
> >
> gtk_widget_set_size_request(appdata.happy,
> 160, 180);
> > appdata.neutral =
> gtk_image_new_from_file("./smileys/neutral.jpg");
> >
> gtk_widget_set_size_request(appdata.neutral,
> 160, 180);
> > appdata.sad =
> gtk_image_new_from_file("./smileys/sad.jpg");
> >
> gtk_widget_set_size_request(appdata.sad,
> 160, 180);
> > appdata.surprise =
> gtk_image_new_from_file("./smileys/surprise.jpg");
> >
> gtk_widget_set_size_request(appdata.surprise,
> 160, 180);
> > appdata.unknown =
> gtk_image_new_from_file("./smileys/unknown.jpg");
> >
> gtk_widget_set_size_request(appdata.unknown,
> 160, 180);
> >
> > appdata.smiley =
> gtk_image_new_from_file("./smileys/unknown.jpg");
> >
> gtk_widget_set_size_request(appdata.smiley,
> 160, 180);
> >
> gtk_box_pack_start(GTK_BOX(vbox_button),
> appdata.smiley, FALSE,
> > FALSE, 0);
> >
> > g_signal_connect(G_OBJECT(button1),
> "clicked",
> > G_CALLBACK(button1_pressed),
> &appdata);
> >
> > g_signal_connect(G_OBJECT(button2),
> "clicked",
> > G_CALLBACK(button2_pressed),
> &appdata);
> >
> >
> >
> gtk_container_add(GTK_CONTAINER(appdata.window),
> hbox);
> >
> > /* Initialize the GTK pipeline */
> > if(!initialize_pipeline(&appdata,
> &argc, &argv))
> > {
> > hildon_banner_show_information(
> > GTK_WIDGET(appdata.window),
> > "gtk-dialog-error",
> > "Failed to initialize
> pipeline");
> > }
> >
> >
> >
> >
> g_signal_connect(G_OBJECT(appdata.window),
> "destroy",
> > G_CALLBACK(destroy_pipeline),
> &appdata);
> >
> >
> > /* Begin the main application */
> > example_gui_run(appdata.program,
> appdata.window);
> >
> > /* Free the gstreamer resources.
> Elements added
> > * to the pipeline will be freed
> automatically */
> >
> > return 0;
> > }
> >
> >
> > What I'd like to do is to modify the
> data_photo buffer to draw a
> > rectangle in it (in the process_frame
> function), and draw the content
> > in the appdata.screen GtkWidget. (by the
> way screen is declared as a
> > GtkWidget * in the appdata structure).
> >
> > Thanks in advance for your help !
> > Bruno
> >
>
> >
> ------------------------------------------------------------------------
>
> >
> >
> -------------------------------------------------------------------------
> > This SF.Net email is sponsored by the
> Moblin Your Move Developer's challenge
> > Build the coolest Linux based applications
> with Moblin SDK & win great prizes
> > Grand prize is a trip for two to an Open
> Source event anywhere in the world
> >
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> <http://moblin-contest.org/redirect.php?banner_id=100&url=/>
> >
> ------------------------------------------------------------------------
> >
> >
> _______________________________________________
> > gstreamer-devel mailing list
> > gstreamer-devel at lists.sourceforge.net
> <mailto:gstreamer-devel at lists.sourceforge.net>
> >
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
> >
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin
> Your Move Developer's challenge
> Build the coolest Linux based applications
> with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open
> Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> <http://moblin-contest.org/redirect.php?banner_id=100&url=/>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> <mailto:gstreamer-devel at lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin
> Your Move Developer's challenge
> Build the coolest Linux based applications with
> Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source
> event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> <http://moblin-contest.org/redirect.php?banner_id=100&url=/>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> <mailto:gstreamer-devel at lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your
> Move Developer's challenge
> Build the coolest Linux based applications with
> Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source
> event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> <http://moblin-contest.org/redirect.php?banner_id=100&url=/>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> <mailto:gstreamer-devel at lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move
> Developer's challenge
> Build the coolest Linux based applications with Moblin
> SDK & win great prizes
> Grand prize is a trip for two to an Open Source event
> anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> <http://moblin-contest.org/redirect.php?banner_id=100&url=/>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> <mailto:gstreamer-devel at lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move
> Developer's challenge
> Build the coolest Linux based applications with Moblin SDK &
> win great prizes
> Grand prize is a trip for two to an Open Source event
> anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> <http://moblin-contest.org/redirect.php?banner_id=100&url=/>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> <mailto:gstreamer-devel at lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move
> Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere
> in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> <http://moblin-contest.org/redirect.php?banner_id=100&url=/>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> <mailto:gstreamer-devel at lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's
> challenge
> Build the coolest Linux based applications with Moblin SDK & win
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> <http://moblin-contest.org/redirect.php?banner_id=100&url=/>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> <mailto:gstreamer-devel at lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
More information about the gstreamer-devel
mailing list