opencv and g_main_loop_run(loop) render problem
Erickson.LU
ericksonlushan at gmail.com
Sat Apr 21 16:08:27 UTC 2018
I tried to use appsink get frame and convert it cv::mat and use imshow show
the mat on screen.
The program some part is from opensource code.
I have 2 question :
(1)I think gst_buffer_ref(buffer); used for releasing the Buffer. But why i
can't use gst_buffer_unref(buffer); ???
(2) The program use appsink pull frame Buffer from v4l2src and convert it to
cv::mat, at last use imshow show the mat.
But when i comment out *g_main_loop_thread_ = boost::thread(RunMainLoop); *
Everything is ok . The mat will continuously show the mat, and the log
"frame count = ..." will scroll .
The log is :
*main thread lwpid = 20051
main thread tid = 1807194688
Process_frame thread lwpid = 20055
Process_frame thread tid = 1377380096
frame count = 1
Element APP_SINK0 changed state from NULL to READY.
Element capsfilter1 changed state from NULL to READY.
Element videoconvert0 changed state from NULL to READY.
Element capsfilter0 changed state from NULL to READY.
Element v4l2src0 changed state from NULL to READY.
Element pipeline0 changed state from NULL to READY.
Element capsfilter1 changed state from READY to PAUSED.
Element videoconvert0 changed state from READY to PAUSED.
Element capsfilter0 changed state from READY to PAUSED.
STREAM STATUS received from element v4l2src0: CREATE
Element v4l2src0 changed state from READY to PAUSED.
Element pipeline0 changed state from READY to PAUSED.
Unparsed message received of type: new-clock
Element capsfilter1 changed state from PAUSED to PLAYING.
Element videoconvert0 changed state from PAUSED to PLAYING.
Element capsfilter0 changed state from PAUSED to PLAYING.
Element v4l2src0 changed state from PAUSED to PLAYING.
STREAM STATUS received from element v4l2src0: ENTER
Unparsed message received of type: stream-start
Element APP_SINK0 changed state from READY to PAUSED.
Unparsed message received of type: async-done
Element APP_SINK0 changed state from PAUSED to PLAYING.
Element pipeline0 changed state from PAUSED to PLAYING.
frame count = 2
frame count = 3
frame count = 4
frame count = 5
frame count = 6
frame count = 7
frame count = 8
frame count = 9
frame count = 10
...*
But when I don't comment out *g_main_loop_thread_ =
boost::thread(RunMainLoop); *
The mat just show only one frame on my screen and pause at there. And the
log is:
*main thread lwpid = 18025
main thread tid = 4097198656
RunningMainLoop thread lwpid = 18027
RunningMainLoop thread tid = 3675776768
Element APP_SINK0 changed state from NULL to READY.
Element capsfilter1 changed state from NULL to READY.
Element videoconvert0 changed state from NULL to READY.
Element capsfilter0 changed state from NULL to READY.
Element v4l2src0 changed state from NULL to READY.
Element pipeline0 changed state from NULL to READY.
Element capsfilter1 changed state from READY to PAUSED.
Element videoconvert0 changed state from READY to PAUSED.
Element capsfilter0 changed state from READY to PAUSED.
STREAM STATUS received from element v4l2src0: CREATE
Element v4l2src0 changed state from READY to PAUSED.
Element pipeline0 changed state from READY to PAUSED.
STREAM STATUS received from element v4l2src0: ENTER
Unparsed message received of type: new-clock
Process_frame thread lwpid = 18029
Element capsfilter1 changed state from PAUSED to PLAYING.
Process_frame thread tid = 3658991360
Unparsed message received of type: stream-start
Element videoconvert0 changed state from PAUSED to PLAYING.
Element capsfilter0 changed state from PAUSED to PLAYING.
Element v4l2src0 changed state from PAUSED to PLAYING.
Element APP_SINK0 changed state from READY to PAUSED.Unparsed message
received of type: async-done
Element APP_SINK0 changed state from PAUSED to PLAYING.
frame count = 1
Element pipeline0 changed state from PAUSED to PLAYING.
test_now: Fatal IO error 11 (Resource temporarily unavailable) on X server
:0.*
If i use *RunMainLoop(); instead of g_main_loop_thread_ =
boost::thread(RunMainLoop);* on the main thread .
It will block at *g_main_loop_run(loop);*. I can understand at blocking at
there. But why i run it at a new thread it will block the Process_frame
method at cv::waitKey(30); ? Does it the function *g_main_loop_run* will
occupy the X Windows server or something, make opencv can't use that ?
The whole program is bleow:
GstElement* pipeline;
GstBus* bus;
boost::thread g_main_loop_thread_;
GstElement* appsink;
boost::mutex mutex;
GstSample* retrievedBuffer;
GstSample* currentBuffer;
#define SINK0 "v4l2src device=/dev/video0 ! video/x-raw,format=(string)I420,
width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1 !
videoconvert ! video/x-raw,format=(string)BGR, width=(int)640,
height=(int)480, pixel-aspect-ratio=(fraction)1/1 ! appsink name=APP_SINK0
caps=video/x-raw,format=BGR,width=640,height=480,pixel-aspect-ratio=1/1"
const cv::Scalar RED(0,0,255), GREEN(0,255,0);
void DoNewSample()
{
GstSample* sample = gst_app_sink_pull_sample(GST_APP_SINK(appsink));
if (sample) {
boost::lock_guard<boost::mutex> guard(mutex);
if (currentBuffer != 0)
{
gst_sample_unref(currentBuffer);
}
currentBuffer = sample;
}
}
bool IsNewFrameAvailable()
{
boost::lock_guard<boost::mutex> guard(mutex);
return ((retrievedBuffer == 0) && (currentBuffer != 0));
}
bool GetSample(GstSample** sample)
{
boost::lock_guard<boost::mutex> guard(mutex);
if (retrievedBuffer == 0) {
if (currentBuffer != 0) {
retrievedBuffer = currentBuffer;
currentBuffer = 0;
(*sample) = retrievedBuffer;
retrievedBuffer=0;
}
} else {
(*sample) = NULL;
return false;
}
return true;
}
void ReleaseSample()
{
boost::lock_guard<boost::mutex> guard(mutex);
if (retrievedBuffer)
gst_sample_unref(retrievedBuffer);
retrievedBuffer = 0;
}
void Process_frame(){
GstSample* sample;
GstMapInfo map;
GstStructure* s;
GstBuffer* buffer;
GstCaps* caps;
GstMemory* mem;
int height, width, size;
int count=0;
void* map_data;
g_print("Process_frame thread lwpid = %u\n", syscall(SYS_gettid));
g_print("Process_frame thread tid = %u\n", pthread_self());
while(true){
if(IsNewFrameAvailable()){
GetSample(&sample);
caps = gst_sample_get_caps(sample);
buffer = gst_sample_get_buffer(sample);
s = gst_caps_get_structure(caps, 0);
gst_structure_get_int(s, "height", &height);
gst_structure_get_int(s, "width", &width);
size = gst_buffer_get_size(buffer);
if (gst_buffer_map (buffer, &map, GST_MAP_READ)) {
cv::Mat frame(height, width, CV_8UC3,map.data,
cv::Mat::AUTO_STEP);
std::string msg ="Copy Right";
cv::Point origin(0,0+13);
cv::putText( frame, msg, origin, 1, 1, RED);
count++;
g_print("frame count = %d\n",count);
cv::imshow("View_CV",frame);
cv::waitKey(30);
gst_buffer_unmap (buffer, &map);
gst_buffer_ref(buffer);
}
gst_sample_unref (sample);
ReleaseSample();
} else {
}
}
}
void EndOfStreamCallback(GstAppSink* appsink, gpointer user_data)
{
}
GstFlowReturn PrerollCallback(GstAppSink* appsink, gpointer user_data)
{
GstSample* sample = gst_app_sink_pull_preroll(appsink);
gst_sample_unref(sample);
return GST_FLOW_OK;
}
GstFlowReturn SampleCallback(GstAppSink* appsink, gpointer user_data)
{
DoNewSample();
return GST_FLOW_OK;
}
bool GstMessageParser(GstBus* bus, GstMessage* msg, pipelinewrapper*
pipeline)
{
if (msg != NULL) {
GError* err = 0;
gchar* debug_info = 0;
switch (GST_MESSAGE_TYPE(msg)) {
case GST_MESSAGE_ERROR:
gst_message_parse_error(msg, &err, &debug_info);
g_printerr("Error received from element %s: %s\n",
GST_OBJECT_NAME(msg->src), err->message);
g_printerr("Debugging information: %s\n", debug_info ?
debug_info : "none");
g_clear_error(&err);
g_free(debug_info);
break;
case GST_MESSAGE_WARNING:
gst_message_parse_warning(msg, &err, &debug_info);
g_printerr("Warning received from element %s: %s\n",
GST_OBJECT_NAME(msg->src), err->message);
g_printerr("Debugging information: %s\n", debug_info ?
debug_info : "none");
g_clear_error(&err);
g_free(debug_info);
break;
case GST_MESSAGE_INFO:
gst_message_parse_info(msg, &err, &debug_info);
g_printerr("Info received from element %s: %s\n",
GST_OBJECT_NAME(msg->src), err->message);
g_printerr("Debugging information: %s\n", debug_info ?
debug_info : "none");
g_clear_error(&err);
g_free(debug_info);
break;
case GST_MESSAGE_EOS:
g_print("End-Of-Stream reached.\n");
break;
case GST_MESSAGE_STATE_CHANGED:
GstState old_state, new_state;
gst_message_parse_state_changed(msg, &old_state, &new_state,
0);
g_print("Element %s changed state from %s to %s.\n",
GST_OBJECT_NAME(msg->src), gst_element_state_get_name(old_state),
gst_element_state_get_name(new_state));
break;
case GST_MESSAGE_QOS:
break;
case GST_MESSAGE_STREAM_STATUS:
GstStreamStatusType stream_status_type;
GstElement* owner;
const gchar* stream_status_type_string;
gst_message_parse_stream_status(msg, &stream_status_type,
&owner);
switch (stream_status_type) {
case GST_STREAM_STATUS_TYPE_CREATE:
stream_status_type_string = "CREATE";
break;
case GST_STREAM_STATUS_TYPE_ENTER:
stream_status_type_string = "ENTER";
break;
case GST_STREAM_STATUS_TYPE_LEAVE:
stream_status_type_string = "LEAVE";
break;
case GST_STREAM_STATUS_TYPE_DESTROY:
stream_status_type_string = "DESTROY";
break;
case GST_STREAM_STATUS_TYPE_START:
stream_status_type_string = "START";
break;
case GST_STREAM_STATUS_TYPE_PAUSE:
stream_status_type_string = "PAUSE";
break;
case GST_STREAM_STATUS_TYPE_STOP:
stream_status_type_string = "STOP";
break;
}
g_printerr("STREAM STATUS received from element %s: %s\n",
GST_OBJECT_NAME(owner), stream_status_type_string);
//g_free (stream_status_type_string);
break;
default:
g_printerr("Unparsed message received of type: %s\n",
gst_message_type_get_name(GST_MESSAGE_TYPE(msg)));
break;
}
}
return true;
}
void RunMainLoop()
{
GMainLoop* loop = g_main_loop_new(NULL, FALSE);
g_print("RunningMainLoop thread lwpid = %u\n", syscall(SYS_gettid));
g_print("RunningMainLoop thread tid = %u\n", pthread_self());
g_main_loop_run(loop);
}
int
main (int argc, char *argv[])
{
gst_init(&argc, &argv);
// XInitThreads();
boost::thread g_main_loop_thread0;
GstAppSinkCallbacks appsinkCallbacks;
std::string pipelineString = SINK0;
std::string appsink_name="APP_SINK0";
GError* err = 0;
pipeline = gst_parse_launch(pipelineString.c_str(), &err);
bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
gst_bus_add_watch(bus, (GstBusFunc)GstMessageParser, NULL);
g_main_loop_thread_ = boost::thread(RunMainLoop);
// RunMainLoop();
appsink = gst_bin_get_by_name(GST_BIN(pipeline), appsink_name.c_str());
g_print("main thread lwpid = %u\n", syscall(SYS_gettid));
g_print("main thread tid = %u\n", pthread_self());
appsinkCallbacks.new_preroll = &PrerollCallback;
appsinkCallbacks.new_sample = &SampleCallback;
// appsinkCallbacks.eos = &EndOfStreamCallback;
gst_app_sink_set_drop(GST_APP_SINK(appsink), true);
gst_app_sink_set_max_buffers(GST_APP_SINK(appsink), 1);
gst_app_sink_set_callbacks(GST_APP_SINK(appsink), &appsinkCallbacks,
NULL, NULL);
GstStateChangeReturn ret = gst_element_set_state(pipeline,
GST_STATE_PLAYING);
if (ret == GST_STATE_CHANGE_FAILURE) {
g_printerr("Unable to set the pipeline to the playing state");
gst_object_unref(pipeline);
return false;
}
// Process_frame();
g_main_loop_thread0 = boost::thread(&Process_frame);
g_main_loop_thread0.join();
/* cleanup and exit */
gst_element_set_state(pipeline,GST_STATE_NULL);
gst_object_unref (pipeline);
exit (0);
}
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
More information about the gstreamer-devel
mailing list