Conversion of cv::Mat to GstBuffer
Jan Schmidt
thaytan at noraisin.net
Mon Mar 25 13:48:32 UTC 2019
Hi,
On 25/3/19 10:36 pm, Saurabh Bora wrote:
> Thanks Michael,
> It seems there is some problem with the input image itself. I tried
> different image of 640*480 resolution and code worked just fine.
For frame sizes that aren't a multiple of 4 bytes, you need to round up
and add padding - GStreamer buffers have an implicit stride that's the
nearest multiple of 4. Alternatively, you can add a GstVideoMeta to the
GstBuffer describing the actual stride.
Cheers,
Jan.
>
> Regards,
> Saurabh Bora
>
> On Fri, Mar 15, 2019 at 2:52 AM Michael Gruner
> <michael.gruner at ridgerun.com <mailto:michael.gruner at ridgerun.com>> wrote:
>
> Can you run your app with warning logging enabled? Any of the
> following should work:
>
> ./app --gst-debug=2
>
> Or
>
> GST_DEBUG=2 ./app
>
>
>
>> On Mar 14, 2019, at 2:17 PM, Saurabh Bora <saurabh9bora at gmail.com
>> <mailto:saurabh9bora at gmail.com>> wrote:
>>
>> Hi Experts!,
>>
>> Thanks!! Michael, I tried the approach suggested by you and it
>> works fine. I could verify the conversion of cv::Mat to GstBuffer
>> and vice versa. On cv::write I get exact image that I had copied
>> to GstBuffer.
>>
>> However, I am unable to get correct image on autovidesink. With
>> current code below, I see a skewed and distorted grayscale image.
>> Note that I had to set width and height of videocaps on appsrc
>> element as 133, 133 respectively, even though dimesnions of my
>> input image is 134*134. If I set width and height as 134, 134 in
>> videocaps, I see a black videosink.
>>
>> //----------------------------------------------------------
>> #include <gst/gst.h>
>> #include <opencv2/opencv.hpp>
>> #include <opencv2/highgui.hpp>
>>
>> static GMainLoop *loop;
>>
>> void buffer_destroy(gpointer data)
>> {
>> cv::Mat* done = (cv::Mat*)data;
>> delete done;
>> }
>>
>> static void
>> cb_need_data(GstElement *appsrc,
>> guint unused_size,
>> gpointer user_data)
>> {
>> static GstClockTime timestamp = 0;
>> GstBuffer *buffer;
>> GstFlowReturn ret;
>> char *data1;
>> GstMapInfo map;
>>
>> cv::Mat* img = new
>> cv::Mat(cv::imread("C:/Users/212730892.LOGON/Desktop/lena.jpg"));
>>
>> std::cout << "cb_need_data called" << std::endl;
>>
>> gsize sizeInBytes = img->total() * img->elemSize();
>> std::cout << "Size getting copied: " << sizeInBytes << std::endl;
>> buffer = gst_buffer_new_wrapped_full((GstMemoryFlags)0,
>> (gpointer)(img->data), sizeInBytes, 0, sizeInBytes,
>> (gpointer)img, (GDestroyNotify)buffer_destroy);
>>
>> gsize bufferSize = gst_buffer_get_size(buffer);
>> std::cout << "Buffer Size: " << bufferSize << std::endl;
>>
>> //Convert GstBuffer back to cv::Mat, to check if data was
>> correctly copied
>> GstMapInfo info;
>> gst_buffer_map(buffer, &info, GST_MAP_READ);
>> cv::Mat img2(cv::Size(img->size().height, img->size().width),
>> CV_8UC3, (char*)(info.data));
>> cv::imwrite("C:/Users/212730892.LOGON/Desktop/dummy1.jpg", img2);
>>
>> GST_BUFFER_OFFSET(buffer) = 0;
>> GST_BUFFER_PTS(buffer) = timestamp;
>> GST_BUFFER_DURATION(buffer) = gst_util_uint64_scale_int(1,
>> GST_SECOND, 30);
>> //GST_BUFFER_DURATION(buffer) = GST_CLOCK_TIME_NONE;
>>
>> timestamp += GST_BUFFER_DURATION(buffer);
>>
>> g_signal_emit_by_name(appsrc, "push-buffer", buffer, &ret);
>> gst_buffer_unref(buffer);
>>
>> if (ret != GST_FLOW_OK) {
>> /* something wrong, stop pushing */
>> g_main_loop_quit(loop);
>> }
>> }
>>
>> gint
>> main(gint argc,
>> gchar *argv[])
>> {
>> GstElement *pipeline, *appsrc, *conv, *videosink;
>>
>> /* init GStreamer */
>> gst_init(&argc, &argv);
>> loop = g_main_loop_new(NULL, FALSE);
>>
>> /* setup pipeline */
>> pipeline = gst_pipeline_new("pipeline");
>> appsrc = gst_element_factory_make("appsrc", "source");
>> conv = gst_element_factory_make("videoconvert", "conv");
>> videosink = gst_element_factory_make("autovideosink", "videosink");
>>
>> /* setup */
>> g_object_set(G_OBJECT(appsrc), "caps",
>> gst_caps_new_simple("video/x-raw",
>> "format", G_TYPE_STRING, "RGB",
>> "width", G_TYPE_INT, 133,
>> "height", G_TYPE_INT, 133,
>> "framerate", GST_TYPE_FRACTION, 1, 30,
>> NULL), NULL);
>> gst_bin_add_many(GST_BIN(pipeline), appsrc, conv, videosink, NULL);
>> gst_element_link_many(appsrc, conv, videosink, NULL);
>>
>> /* setup appsrc */
>> g_object_set(G_OBJECT(appsrc),
>> "stream-type", 0,
>> "format", GST_FORMAT_TIME, NULL);
>> g_signal_connect(appsrc, "need-data", G_CALLBACK(cb_need_data),
>> NULL);
>>
>> /* play */
>> gst_element_set_state(pipeline, GST_STATE_PLAYING);
>> g_main_loop_run(loop);
>>
>> if (gst_element_set_state(pipeline, GST_STATE_NULL) ==
>> GST_STATE_CHANGE_FAILURE)
>> {
>> g_printerr("Unable to set the pipeline to the playing state.\n");
>> }
>>
>> // Free resources
>> gst_object_unref(GST_OBJECT(pipeline));
>> g_main_loop_unref(loop);
>>
>> return 0;
>> }
>> //-------------------------------------------------------------------------
>>
>> I am new to gstreamer framework and it would be a great help if
>> someone could point me in right direction.
>>
>> Thanks and Regards,
>> Saurabh Bora
>>
>>
>> On Wed, Mar 13, 2019 at 11:48 PM Michael Gruner
>> <michael.gruner at ridgerun.com
>> <mailto:michael.gruner at ridgerun.com>> wrote:
>>
>> It looks like img2, being a local variable, ends its life
>> when “cb_need_data” ends. That leaves the pointer in “buffer”
>> invalid. For this one you’re better off with the copy.
>>
>> Another option is to alloc img2 on the heap, and use
>> gst_buffer_new_wrapped_full to carry img2 along with buffer
>> and free then together.
>>
>> Michael
>> www.ridgerun.com <http://www.ridgerun.com/>
>>
>> On Mar 13, 2019, at 10:51 AM, Saurabh Bora
>> <saurabh9bora at gmail.com <mailto:saurabh9bora at gmail.com>> wrote:
>>
>>> Hi Michael,
>>>
>>> Thanks! for your input. I tried what you suggested as below:
>>>
>>> //---------------------------------------------------------------
>>> #include <gst/gst.h>
>>> #include <opencv2/opencv.hpp>
>>> #include <opencv2/highgui.hpp>
>>> #include <opencv2/imgproc/imgproc.hpp>
>>>
>>> static GMainLoop *loop;
>>>
>>> static void
>>> cb_need_data(GstElement *appsrc,
>>> guint unused_size,
>>> gpointer user_data)
>>> {
>>> static GstClockTime timestamp = 0;
>>> GstBuffer *buffer;
>>> guint size, depth, height, width, step, channels;
>>> GstFlowReturn ret;
>>> guchar *data1;
>>> GstMapInfo map;
>>>
>>> cv::Mat img2;
>>> cv::Mat img =
>>> cv::imread("C:/Users/212730892.LOGON/Desktop/lena.jpg");
>>> cv::cvtColor(img, img2, cv::COLOR_BGR2RGB);
>>>
>>> height = img.size().height;
>>> width = img.size().width;
>>> //step = img.widthStep;
>>> channels = img.channels();
>>> //depth = img->depth;
>>> data1 = (guchar *)img.data;
>>> size = height * width * channels * sizeof(guchar);
>>>
>>> std::cout << "cb_need_data called" << std::endl;
>>>
>>> buffer = gst_buffer_new_wrapped(img2.data, img2.total() *
>>> img2.elemSize());
>>>
>>> GstMapInfo info;
>>> gst_buffer_map(buffer, &info, GST_MAP_READ);
>>> cv::Mat mat(height, width, CV_8UC3, info.data);
>>> cv::imshow("Output", mat);
>>> //gst_buffer_unmap(buffer, &info);
>>>
>>> GST_BUFFER_PTS(buffer) = timestamp;
>>> GST_BUFFER_DURATION(buffer) = gst_util_uint64_scale_int(1,
>>> GST_SECOND, 2);
>>>
>>> timestamp += GST_BUFFER_DURATION(buffer);
>>>
>>> g_signal_emit_by_name(appsrc, "push-buffer", buffer, &ret);
>>> //gst_buffer_unref(buffer);
>>>
>>> if (ret != GST_FLOW_OK) {
>>> /* something wrong, stop pushing */
>>> g_main_loop_quit(loop);
>>> }
>>> }
>>>
>>> gint
>>> main(gint argc,
>>> gchar *argv[])
>>> {
>>> GstElement *pipeline, *appsrc, *conv, *videosink;
>>>
>>> /* init GStreamer */
>>> gst_init(&argc, &argv);
>>> loop = g_main_loop_new(NULL, FALSE);
>>>
>>> /* setup pipeline */
>>> pipeline = gst_pipeline_new("pipeline");
>>> appsrc = gst_element_factory_make("appsrc", "source");
>>> conv = gst_element_factory_make("videoconvert", "conv");
>>> videosink = gst_element_factory_make("autovideosink",
>>> "videosink");
>>>
>>> /* setup */
>>> g_object_set(G_OBJECT(appsrc), "caps",
>>> gst_caps_new_simple("video/x-raw",
>>> "format", G_TYPE_STRING, "RGB",
>>> "width", G_TYPE_INT, 134,
>>> "height", G_TYPE_INT, 134,
>>> "framerate", GST_TYPE_FRACTION, 1, 1,
>>> NULL), NULL);
>>> gst_bin_add_many(GST_BIN(pipeline), appsrc, conv, videosink,
>>> NULL);
>>> gst_element_link_many(appsrc, conv, videosink, NULL);
>>> //g_object_set (videosink, "device", "/dev/video0", NULL);
>>> /* setup appsrc */
>>> g_object_set(G_OBJECT(appsrc),
>>> "stream-type", 0,
>>> "format", GST_FORMAT_TIME, NULL);
>>> g_signal_connect(appsrc, "need-data",
>>> G_CALLBACK(cb_need_data), NULL);
>>>
>>> /* play */
>>> gst_element_set_state(pipeline, GST_STATE_PLAYING);
>>> g_main_loop_run(loop);
>>>
>>> if (gst_element_set_state(pipeline, GST_STATE_NULL) ==
>>> GST_STATE_CHANGE_FAILURE)
>>> {
>>> g_printerr("Unable to set the pipeline to the playing
>>> state.\n");
>>> }
>>>
>>> // Free resources
>>> gst_object_unref(GST_OBJECT(pipeline));
>>> g_main_loop_unref(loop);
>>>
>>> return 0;
>>> }
>>> //----------------------------------------------------------------------------------------------
>>>
>>> It still does not work as expected. It would be a great help
>>> if you could point out where I am going wrong.
>>> Thanks!!
>>>
>>> On Wed, Mar 13, 2019 at 7:40 PM Michael Gruner
>>> <michael.gruner at ridgerun.com
>>> <mailto:michael.gruner at ridgerun.com>> wrote:
>>>
>>> Hi Saurabh
>>>
>>> Yo may convert from a cv::Mat to a GstBuffer without a
>>> memory copy by using gst_buffer_new_wrapped:
>>>
>>> https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstBuffer.html#gst-buffer-new-wrapped
>>>
>>> Something like:
>>> buffer = gst_buffer_new_wrapped (mat.data,
>>> mat.total()*mat.elemSize());
>>>
>>> In order to make a cv::Mat from a GstBuffer you first
>>> need to take the data out of the buffer.
>>>
>>> GstMapInfo info;
>>> gst_buffer_map (buffer, &info, GST_MAP_READ); // or write
>>> mat = cv::Mat(height, width, CV_8C3, map.data); //
>>> change your format accordingly
>>> ...
>>> gst_buffer_unmap (buffer, &info);
>>>
>>> Hope it helps.
>>>
>>> Michael
>>> www.ridgerun.com <http://www.ridgerun.com/>
>>>
>>> On Mar 13, 2019, at 7:15 AM, Saurabh Bora
>>> <saurabh9bora at gmail.com <mailto:saurabh9bora at gmail.com>>
>>> wrote:
>>>
>>>> Hi Experts!!,
>>>>
>>>> I am trying to read opencv images locally and push them
>>>> into gstreamer pipeline. I am using appsrc element with
>>>> "need-data" signal attached to call-back function.
>>>> Please look at the code below.
>>>>
>>>> //------------------------------------------------------------------------------------------------------
>>>> #include <gst/gst.h>
>>>> #include <opencv2/opencv.hpp>
>>>> #include <opencv2/highgui.hpp>
>>>>
>>>> static GMainLoop *loop;
>>>>
>>>> static void
>>>> cb_need_data(GstElement *appsrc,
>>>> guint unused_size,
>>>> gpointer user_data)
>>>> {
>>>> static GstClockTime timestamp = 0;
>>>> GstBuffer *buffer;
>>>> guint size, depth, height, width, step, channels;
>>>> GstFlowReturn ret;
>>>> guchar *data1;
>>>> GstMapInfo map;
>>>>
>>>> cv::Mat img =
>>>> cv::imread("C:/Users/212730892.LOGON/Desktop/lena.jpg");
>>>> height = img.size().height;
>>>> width = img.size().width;
>>>> //step = img.widthStep;
>>>> channels = img.channels();
>>>> //depth = img->depth;
>>>> data1 = (guchar *)img.data;
>>>> size = height * width* channels;
>>>>
>>>> // Copy cv::Mat to GstBuffer
>>>> buffer = gst_buffer_new_allocate(NULL, size, NULL);
>>>> gst_buffer_map(buffer, &map, GST_MAP_WRITE);
>>>> memcpy((guchar *)map.data, data1,
>>>> gst_buffer_get_size(buffer));
>>>>
>>>> //Convert GstBuffer back to cv::Mat and write it, to
>>>> check if data was correctly copied. //This is where I
>>>> get to know that data is not correct.
>>>> cv::Mat img2(cv::Size(134, 134), CV_8UC3, (char*)(buffer));
>>>> cv::imwrite("C:/Users/212730892.LOGON/Desktop/dummy1.jpg",
>>>> img2);
>>>>
>>>> GST_BUFFER_PTS(buffer) = timestamp;
>>>> GST_BUFFER_DURATION(buffer) =
>>>> gst_util_uint64_scale_int(1, GST_SECOND, 2);
>>>>
>>>> timestamp += GST_BUFFER_DURATION(buffer);
>>>>
>>>> g_signal_emit_by_name(appsrc, "push-buffer", buffer, &ret);
>>>> //gst_buffer_unref(buffer);
>>>>
>>>> if (ret != GST_FLOW_OK) {
>>>> /* something wrong, stop pushing */
>>>> g_main_loop_quit(loop);
>>>> }
>>>> }
>>>>
>>>> gint
>>>> main(gint argc,
>>>> gchar *argv[])
>>>> {
>>>> GstElement *pipeline, *appsrc, *conv, *videosink;
>>>>
>>>> /* init GStreamer */
>>>> gst_init(&argc, &argv);
>>>> loop = g_main_loop_new(NULL, FALSE);
>>>>
>>>> /* setup pipeline */
>>>> pipeline = gst_pipeline_new("pipeline");
>>>> appsrc = gst_element_factory_make("appsrc", "source");
>>>> conv = gst_element_factory_make("videoconvert", "conv");
>>>> videosink = gst_element_factory_make("autovideosink",
>>>> "videosink");
>>>>
>>>> /* setup */
>>>> g_object_set(G_OBJECT(appsrc), "caps",
>>>> gst_caps_new_simple("video/x-raw",
>>>> "format", G_TYPE_STRING, "RGB",
>>>> "width", G_TYPE_INT, 134,
>>>> "height", G_TYPE_INT, 134,
>>>> "framerate", GST_TYPE_FRACTION, 1, 1,
>>>> NULL), NULL);
>>>> gst_bin_add_many(GST_BIN(pipeline), appsrc, conv,
>>>> videosink, NULL);
>>>> gst_element_link_many(appsrc, conv, videosink, NULL);
>>>> //g_object_set (videosink, "device", "/dev/video0", NULL);
>>>> /* setup appsrc */
>>>> g_object_set(G_OBJECT(appsrc),
>>>> "stream-type", 0,
>>>> "format", GST_FORMAT_TIME, NULL);
>>>> g_signal_connect(appsrc, "need-data",
>>>> G_CALLBACK(cb_need_data), NULL);
>>>>
>>>> /* play */
>>>> gst_element_set_state(pipeline, GST_STATE_PLAYING);
>>>> g_main_loop_run(loop);
>>>>
>>>> if (gst_element_set_state(pipeline, GST_STATE_NULL) ==
>>>> GST_STATE_CHANGE_FAILURE)
>>>> {
>>>> g_printerr("Unable to set the pipeline to the playing
>>>> state.\n");
>>>> }
>>>>
>>>> // Free resources
>>>> gst_object_unref(GST_OBJECT(pipeline));
>>>> g_main_loop_unref(loop);
>>>>
>>>> return 0;
>>>> }
>>>> //-------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> In cb_need_data, the part where copying I am copying to
>>>> GstBuffer seems to be wrong and it does not work as
>>>> expected.
>>>> Can anyone, please help me understand, how can I copy
>>>> the image data perfectly and pass it downstream to
>>>> other elements without losing any data?
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Saurabh Bora
>>>>
>>>> PH NO : 7038166900
>>>> EMAIL : saurabh9bora at gmail.com
>>>> <mailto:saurabh9bora at gmail.com>
>>>> saurabh9bora at outlook.com
>>>> <mailto:saurabh9bora at outlook.com>
>>>> _______________________________________________
>>>> gstreamer-devel mailing list
>>>> gstreamer-devel at lists.freedesktop.org
>>>> <mailto:gstreamer-devel at lists.freedesktop.org>
>>>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.freedesktop.org
>>> <mailto:gstreamer-devel at lists.freedesktop.org>
>>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Saurabh Bora
>>>
>>> PH NO : 7038166900
>>> EMAIL : saurabh9bora at gmail.com <mailto:saurabh9bora at gmail.com>
>>> saurabh9bora at outlook.com
>>> <mailto:saurabh9bora at outlook.com>
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.freedesktop.org
>>> <mailto:gstreamer-devel at lists.freedesktop.org>
>>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> <mailto:gstreamer-devel at lists.freedesktop.org>
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>>
>> --
>> Thanks and Regards,
>> Saurabh Bora
>>
>> PH NO : 7038166900
>> EMAIL : saurabh9bora at gmail.com <mailto:saurabh9bora at gmail.com>
>> saurabh9bora at outlook.com
>> <mailto:saurabh9bora at outlook.com>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> <mailto:gstreamer-devel at lists.freedesktop.org>
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> <mailto:gstreamer-devel at lists.freedesktop.org>
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
>
> --
> Thanks and Regards,
> Saurabh Bora
>
> PH NO : 7038166900
> EMAIL : saurabh9bora at gmail.com <mailto:saurabh9bora at gmail.com>
> saurabh9bora at outlook.com <mailto:saurabh9bora at outlook.com>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20190326/d5601df0/attachment-0001.html>
More information about the gstreamer-devel
mailing list