Accessing video RGB data

[Stefan Sauer]

On 10/06/2011 11:20 PM, Sami Pietila wrote:
> Hi,
>
> Thanks for reply! I changed to ximagesink and now I am getting RGB data.
>
> I don't need to modify movie frames, but I need to read them and do
> some calculations. Calculations might take some time, so I need a way
> to control how gstreamers feeds the frames. That is, I need a way to
> ask next frame from gstreamer instead of gstreamer feeding them at a
> constant rate.
>
> How can I get a new frame only when requested?

In your pad_probe just do the calculation as you need. ximagesink will
throw away late frames. A smarter way might be to do:

... ! colorspace ! tee name = t ! queue leaky=upstream ! appsink
                                              t. ! queue ! ximagesink

This way you can use get the frames from appsink (better that using
pad-probes or fakesink with the handoff signal). Also the playback gets
not disturbed and you process the frames on appsink as you manage. All
other frames get dropped.

Stefan

> 2011/10/4 Stefan Sauer <ensonic at hora-obscura.de>:
>> On 10/01/2011 10:16 AM, Sami Pietila wrote:
>>> Hi,
>>>
>>> I am trying to access RGB data of image frames in a mov -file. Below
>>> is code that I have constructed using gstreamer examples and
>>> documentation. cb_have_data() doesn't seem to give me RGB data. I am
>>> setting the data to 0 (=black?), but I can see shortly a green screen
>>> on the video and then comes invalid pointer error.
>> when you see green on all 0, you are in a yuv format and not rgb. In the
>> code below the ffmpegcolorspace wont convert anything as xvimagsink will
>> prefer yuv. If you change the later to ximagesink you'll get RGB.
>>
>>> How can I read RGB data from a video, frame by frame? I need also a
>>> way to control the flow of frames, in a way that once I have done with
>>> processing one frame I could ask gstreamer to give me next frame. Code
>>> examples would be great.
>> If you wants to modify the image data you better write an element.
>>
>> Stefan
>>
>>> Thanks
>>>
>>>
>>> #include <iostream>
>>> #include <gst/gst.h>
>>>
>>> GstElement *pipeline, *video;
>>>
>>> static gboolean
>>> cb_have_data (GstPad *pad, GstBuffer *buffer, gpointer u_data) {
>>>
>>>   guint16 *data = (guint16 *) GST_BUFFER_DATA (buffer);
>>>   GstCaps *caps = gst_buffer_get_caps(buffer);
>>>
>>>   char m_width[] = "width";
>>>   char m_height[] = "height";
>>>
>>>   GstStructure *structure = gst_caps_get_structure(caps, 0);
>>>
>>>   const GValue *width_gval = gst_structure_get_value(structure, m_width);
>>>   const GValue *height_gval = gst_structure_get_value(structure, m_height);
>>>
>>>   int width = g_value_get_int (width_gval);
>>>   int height = g_value_get_int(height_gval);
>>>
>>>   for (int y = 0; y < height; y++) {
>>>         for (int x = 0; x < width; x++) {
>>>                 data[width*y + x] = 0;
>>>         }
>>>   }
>>>
>>>   return TRUE;
>>> }
>>>
>>> static void
>>> cb_newpad (GstElement *decodebin,
>>>          GstPad     *pad,
>>>          gboolean    last,
>>>          gpointer    data)
>>> {
>>>   GstCaps *caps;
>>>   GstStructure *str;
>>>   GstPad *videopad;
>>>
>>>   videopad = gst_element_get_static_pad (video, "sink");
>>>   if (GST_PAD_IS_LINKED (videopad)) {
>>>     g_object_unref (videopad);
>>>     return;
>>>   }
>>>
>>>   caps = gst_pad_get_caps (pad);
>>>   str = gst_caps_get_structure (caps, 0);
>>>   if (!g_strrstr (gst_structure_get_name (str), "video")) {
>>>     gst_caps_unref (caps);
>>>     gst_object_unref (videopad);
>>>     return;
>>>   }
>>>   gst_caps_unref (caps);
>>>
>>>   gst_pad_link (pad, videopad);
>>>
>>>   g_object_unref (videopad);
>>> }
>>>
>>> gint main (gint argc, gchar *argv[])
>>> {
>>>   GMainLoop *loop;
>>>   GstElement *src, *dec, *filter, *cs, *sink;
>>>   GstPad *videopad;
>>>   GstBus *bus;
>>>
>>>   gst_init (&argc, &argv);
>>>   loop = g_main_loop_new (NULL, FALSE);
>>>
>>>   pipeline = gst_pipeline_new ("pipeline");
>>>
>>>   bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
>>>
>>>   src = gst_element_factory_make ("filesrc", "source");
>>>   g_object_set (G_OBJECT (src), "location",
>>> "/home/sammy/src/SamScanPlus/src/testi.mov", NULL);
>>>   dec = gst_element_factory_make ("decodebin", "decoder");
>>>   g_signal_connect (dec, "new-decoded-pad", G_CALLBACK (cb_newpad), NULL);
>>>   gst_bin_add_many (GST_BIN (pipeline), src, dec, NULL);
>>>   gst_element_link (src, dec);
>>>
>>>   video = gst_bin_new ("videobin");
>>>   cs = gst_element_factory_make ("ffmpegcolorspace", "cs");
>>>   videopad = gst_element_get_static_pad (cs, "sink");
>>>   sink = gst_element_factory_make ("xvimagesink", "sink");
>>>   gst_bin_add_many (GST_BIN (video), cs, sink, NULL);
>>>   gst_element_link_many(cs, sink, NULL);
>>>   gst_element_add_pad (video, gst_ghost_pad_new ("sink", videopad));
>>>   gst_object_unref (videopad);
>>>   gst_bin_add (GST_BIN (pipeline), video);
>>>
>>>   GstPad *pad = gst_element_get_pad (cs, "src");
>>>   gst_pad_add_buffer_probe (pad, G_CALLBACK (cb_have_data), NULL);
>>>   gst_object_unref (pad);
>>>
>>>   gst_element_set_state (pipeline, GST_STATE_PLAYING);
>>>   g_main_loop_run (loop);
>>>
>>>   gst_element_set_state (pipeline, GST_STATE_NULL);
>>>   gst_object_unref (GST_OBJECT (pipeline));
>>>
>>>   return 0;
>>> }
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel