opencv and g_main_loop_run(loop) render problem
Nicolas Dufresne
nicolas at ndufresne.ca
Sat Apr 21 17:51:48 UTC 2018
Le sam. 21 avr. 2018 12:12, Erickson.LU <ericksonlushan at gmail.com> a écrit :
> I tried to use appsink get frame and convert it cv::mat and use imshow show
> the mat on screen.
> The program some part is from opensource code.
> I have 2 question :
> (1)I think gst_buffer_ref(buffer); used for releasing the Buffer. But why
> i
> can't use gst_buffer_unref(buffer); ???
>
> (2) The program use appsink pull frame Buffer from v4l2src and convert it
> to
> cv::mat, at last use imshow show the mat.
> But when i comment out *g_main_loop_thread_ = boost::thread(RunMainLoop);
> *
> Everything is ok . The mat will continuously show the mat, and the log
> "frame count = ..." will scroll .
>
>
> The log is :
> *main thread lwpid = 20051
> main thread tid = 1807194688
> Process_frame thread lwpid = 20055
> Process_frame thread tid = 1377380096
> frame count = 1
> Element APP_SINK0 changed state from NULL to READY.
> Element capsfilter1 changed state from NULL to READY.
> Element videoconvert0 changed state from NULL to READY.
> Element capsfilter0 changed state from NULL to READY.
> Element v4l2src0 changed state from NULL to READY.
> Element pipeline0 changed state from NULL to READY.
> Element capsfilter1 changed state from READY to PAUSED.
> Element videoconvert0 changed state from READY to PAUSED.
> Element capsfilter0 changed state from READY to PAUSED.
> STREAM STATUS received from element v4l2src0: CREATE
> Element v4l2src0 changed state from READY to PAUSED.
> Element pipeline0 changed state from READY to PAUSED.
> Unparsed message received of type: new-clock
> Element capsfilter1 changed state from PAUSED to PLAYING.
> Element videoconvert0 changed state from PAUSED to PLAYING.
> Element capsfilter0 changed state from PAUSED to PLAYING.
> Element v4l2src0 changed state from PAUSED to PLAYING.
> STREAM STATUS received from element v4l2src0: ENTER
> Unparsed message received of type: stream-start
> Element APP_SINK0 changed state from READY to PAUSED.
> Unparsed message received of type: async-done
> Element APP_SINK0 changed state from PAUSED to PLAYING.
> Element pipeline0 changed state from PAUSED to PLAYING.
> frame count = 2
> frame count = 3
> frame count = 4
> frame count = 5
> frame count = 6
> frame count = 7
> frame count = 8
> frame count = 9
> frame count = 10
> ...*
>
>
>
>
>
>
>
> But when I don't comment out *g_main_loop_thread_ =
> boost::thread(RunMainLoop); *
> The mat just show only one frame on my screen and pause at there. And the
> log is:
>
>
> *main thread lwpid = 18025
> main thread tid = 4097198656
> RunningMainLoop thread lwpid = 18027
> RunningMainLoop thread tid = 3675776768
> Element APP_SINK0 changed state from NULL to READY.
> Element capsfilter1 changed state from NULL to READY.
> Element videoconvert0 changed state from NULL to READY.
> Element capsfilter0 changed state from NULL to READY.
> Element v4l2src0 changed state from NULL to READY.
> Element pipeline0 changed state from NULL to READY.
> Element capsfilter1 changed state from READY to PAUSED.
> Element videoconvert0 changed state from READY to PAUSED.
> Element capsfilter0 changed state from READY to PAUSED.
> STREAM STATUS received from element v4l2src0: CREATE
> Element v4l2src0 changed state from READY to PAUSED.
> Element pipeline0 changed state from READY to PAUSED.
> STREAM STATUS received from element v4l2src0: ENTER
> Unparsed message received of type: new-clock
> Process_frame thread lwpid = 18029
> Element capsfilter1 changed state from PAUSED to PLAYING.
> Process_frame thread tid = 3658991360
> Unparsed message received of type: stream-start
> Element videoconvert0 changed state from PAUSED to PLAYING.
> Element capsfilter0 changed state from PAUSED to PLAYING.
> Element v4l2src0 changed state from PAUSED to PLAYING.
> Element APP_SINK0 changed state from READY to PAUSED.Unparsed message
> received of type: async-done
>
> Element APP_SINK0 changed state from PAUSED to PLAYING.
> frame count = 1
> Element pipeline0 changed state from PAUSED to PLAYING.
> test_now: Fatal IO error 11 (Resource temporarily unavailable) on X server
> :0.*
>
>
>
> If i use *RunMainLoop(); instead of g_main_loop_thread_ =
> boost::thread(RunMainLoop);* on the main thread .
> It will block at *g_main_loop_run(loop);*. I can understand at blocking at
> there. But why i run it at a new thread it will block the Process_frame
> method at cv::waitKey(30); ? Does it the function *g_main_loop_run* will
> occupy the X Windows server or something, make opencv can't use that ?
>
>
>
>
>
>
> The whole program is bleow:
>
> GstElement* pipeline;
> GstBus* bus;
> boost::thread g_main_loop_thread_;
> GstElement* appsink;
> boost::mutex mutex;
> GstSample* retrievedBuffer;
> GstSample* currentBuffer;
>
> #define SINK0 "v4l2src device=/dev/video0 !
> video/x-raw,format=(string)I420,
> width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1 !
> videoconvert ! video/x-raw,format=(string)BGR, width=(int)640,
> height=(int)480, pixel-aspect-ratio=(fraction)1/1 ! appsink name=APP_SINK0
> caps=video/x-raw,format=BGR,width=640,height=480,pixel-aspect-ratio=1/1"
> const cv::Scalar RED(0,0,255), GREEN(0,255,0);
> void DoNewSample()
> {
> GstSample* sample = gst_app_sink_pull_sample(GST_APP_SINK(appsink));
>
> if (sample) {
> boost::lock_guard<boost::mutex> guard(mutex);
> if (currentBuffer != 0)
> {
> gst_sample_unref(currentBuffer);
> }
> currentBuffer = sample;
> }
> }
> bool IsNewFrameAvailable()
> {
> boost::lock_guard<boost::mutex> guard(mutex);
> return ((retrievedBuffer == 0) && (currentBuffer != 0));
> }
> bool GetSample(GstSample** sample)
> {
> boost::lock_guard<boost::mutex> guard(mutex);
> if (retrievedBuffer == 0) {
> if (currentBuffer != 0) {
> retrievedBuffer = currentBuffer;
> currentBuffer = 0;
> (*sample) = retrievedBuffer;
> retrievedBuffer=0;
> }
> } else {
> (*sample) = NULL;
> return false;
> }
>
> return true;
> }
>
> void ReleaseSample()
> {
> boost::lock_guard<boost::mutex> guard(mutex);
>
> if (retrievedBuffer)
> gst_sample_unref(retrievedBuffer);
> retrievedBuffer = 0;
>
> }
>
> void Process_frame(){
> GstSample* sample;
> GstMapInfo map;
> GstStructure* s;
> GstBuffer* buffer;
> GstCaps* caps;
> GstMemory* mem;
> int height, width, size;
> int count=0;
> void* map_data;
> g_print("Process_frame thread lwpid = %u\n", syscall(SYS_gettid));
> g_print("Process_frame thread tid = %u\n", pthread_self());
> while(true){
> if(IsNewFrameAvailable()){
> GetSample(&sample);
> caps = gst_sample_get_caps(sample);
> buffer = gst_sample_get_buffer(sample);
> s = gst_caps_get_structure(caps, 0);
> gst_structure_get_int(s, "height", &height);
> gst_structure_get_int(s, "width", &width);
> size = gst_buffer_get_size(buffer);
> if (gst_buffer_map (buffer, &map, GST_MAP_READ)) {
> cv::Mat frame(height, width, CV_8UC3,map.data,
> cv::Mat::AUTO_STEP);
> std::string msg ="Copy Right";
> cv::Point origin(0,0+13);
> cv::putText( frame, msg, origin, 1, 1, RED);
> count++;
> g_print("frame count = %d\n",count);
> cv::imshow("View_CV",frame);
> cv::waitKey(30);
>
> gst_buffer_unmap (buffer, &map);
> gst_buffer_ref(buffer);
> }
> gst_sample_unref (sample);
> ReleaseSample();
> } else {
> }
> }
> }
>
>
> void EndOfStreamCallback(GstAppSink* appsink, gpointer user_data)
> {
>
> }
>
> GstFlowReturn PrerollCallback(GstAppSink* appsink, gpointer user_data)
> {
> GstSample* sample = gst_app_sink_pull_preroll(appsink);
> gst_sample_unref(sample);
> return GST_FLOW_OK;
> }
> GstFlowReturn SampleCallback(GstAppSink* appsink, gpointer user_data)
> {
> DoNewSample();
> return GST_FLOW_OK;
> }
>
> bool GstMessageParser(GstBus* bus, GstMessage* msg, pipelinewrapper*
> pipeline)
> {
>
>
> if (msg != NULL) {
> GError* err = 0;
> gchar* debug_info = 0;
>
> switch (GST_MESSAGE_TYPE(msg)) {
> case GST_MESSAGE_ERROR:
> gst_message_parse_error(msg, &err, &debug_info);
> g_printerr("Error received from element %s: %s\n",
> GST_OBJECT_NAME(msg->src), err->message);
> g_printerr("Debugging information: %s\n", debug_info ?
> debug_info : "none");
> g_clear_error(&err);
> g_free(debug_info);
> break;
>
> case GST_MESSAGE_WARNING:
> gst_message_parse_warning(msg, &err, &debug_info);
> g_printerr("Warning received from element %s: %s\n",
> GST_OBJECT_NAME(msg->src), err->message);
> g_printerr("Debugging information: %s\n", debug_info ?
> debug_info : "none");
> g_clear_error(&err);
> g_free(debug_info);
> break;
>
> case GST_MESSAGE_INFO:
> gst_message_parse_info(msg, &err, &debug_info);
> g_printerr("Info received from element %s: %s\n",
> GST_OBJECT_NAME(msg->src), err->message);
> g_printerr("Debugging information: %s\n", debug_info ?
> debug_info : "none");
> g_clear_error(&err);
> g_free(debug_info);
> break;
>
> case GST_MESSAGE_EOS:
> g_print("End-Of-Stream reached.\n");
> break;
>
> case GST_MESSAGE_STATE_CHANGED:
> GstState old_state, new_state;
> gst_message_parse_state_changed(msg, &old_state,
> &new_state,
> 0);
> g_print("Element %s changed state from %s to %s.\n",
> GST_OBJECT_NAME(msg->src), gst_element_state_get_name(old_state),
> gst_element_state_get_name(new_state));
> break;
>
> case GST_MESSAGE_QOS:
> break;
>
> case GST_MESSAGE_STREAM_STATUS:
>
> GstStreamStatusType stream_status_type;
> GstElement* owner;
> const gchar* stream_status_type_string;
> gst_message_parse_stream_status(msg, &stream_status_type,
> &owner);
>
> switch (stream_status_type) {
> case GST_STREAM_STATUS_TYPE_CREATE:
> stream_status_type_string = "CREATE";
> break;
> case GST_STREAM_STATUS_TYPE_ENTER:
> stream_status_type_string = "ENTER";
> break;
> case GST_STREAM_STATUS_TYPE_LEAVE:
> stream_status_type_string = "LEAVE";
> break;
> case GST_STREAM_STATUS_TYPE_DESTROY:
> stream_status_type_string = "DESTROY";
> break;
>
> case GST_STREAM_STATUS_TYPE_START:
> stream_status_type_string = "START";
> break;
> case GST_STREAM_STATUS_TYPE_PAUSE:
> stream_status_type_string = "PAUSE";
> break;
> case GST_STREAM_STATUS_TYPE_STOP:
> stream_status_type_string = "STOP";
> break;
> }
>
> g_printerr("STREAM STATUS received from element %s: %s\n",
> GST_OBJECT_NAME(owner), stream_status_type_string);
> //g_free (stream_status_type_string);
> break;
>
> default:
> g_printerr("Unparsed message received of type: %s\n",
> gst_message_type_get_name(GST_MESSAGE_TYPE(msg)));
> break;
> }
> }
> return true;
> }
>
> void RunMainLoop()
> {
> GMainLoop* loop = g_main_loop_new(NULL, FALSE);
>
This is wrong, Null mean default main context, which must be created on the
same thread. It's really hard to control in which thread the one will be
created. In your case, it probably get created in your main function, when
you do gst_bus_add_watch(). You could try to create it here, but in
general, it's more robust to create a new one. Nd then make everything
explicit. Using push_thread_default, and also gst_bus_create_watch and
g_source_attach
g_print("RunningMainLoop thread lwpid = %u\n", syscall(SYS_gettid));
> g_print("RunningMainLoop thread tid = %u\n", pthread_self());
> g_main_loop_run(loop);
> }
>
> int
> main (int argc, char *argv[])
> {
>
> gst_init(&argc, &argv);
> // XInitThreads();
> boost::thread g_main_loop_thread0;
>
> GstAppSinkCallbacks appsinkCallbacks;
> std::string pipelineString = SINK0;
> std::string appsink_name="APP_SINK0";
> GError* err = 0;
> pipeline = gst_parse_launch(pipelineString.c_str(), &err);
>
> bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
> gst_bus_add_watch(bus, (GstBusFunc)GstMessageParser, NULL);
>
> g_main_loop_thread_ = boost::thread(RunMainLoop);
> // RunMainLoop();
> appsink = gst_bin_get_by_name(GST_BIN(pipeline), appsink_name.c_str());
>
> g_print("main thread lwpid = %u\n", syscall(SYS_gettid));
> g_print("main thread tid = %u\n", pthread_self());
>
>
> appsinkCallbacks.new_preroll = &PrerollCallback;
> appsinkCallbacks.new_sample = &SampleCallback;
> // appsinkCallbacks.eos = &EndOfStreamCallback;
> gst_app_sink_set_drop(GST_APP_SINK(appsink), true);
> gst_app_sink_set_max_buffers(GST_APP_SINK(appsink), 1);
> gst_app_sink_set_callbacks(GST_APP_SINK(appsink), &appsinkCallbacks,
> NULL, NULL);
>
>
> GstStateChangeReturn ret = gst_element_set_state(pipeline,
> GST_STATE_PLAYING);
>
> if (ret == GST_STATE_CHANGE_FAILURE) {
> g_printerr("Unable to set the pipeline to the playing state");
> gst_object_unref(pipeline);
> return false;
> }
>
>
> // Process_frame();
> g_main_loop_thread0 = boost::thread(&Process_frame);
> g_main_loop_thread0.join();
>
> /* cleanup and exit */
>
> gst_element_set_state(pipeline,GST_STATE_NULL);
> gst_object_unref (pipeline);
>
> exit (0);
> }
>
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180421/891cdd5a/attachment-0001.html>
More information about the gstreamer-devel
mailing list