Gstreamer with multithread app (Qt)

[George Kiagiadakis]

Hello,

On Wed, Apr 13, 2011 at 12:34 AM, Matias Hernandez Arellano
<msdark at archlinux.cl> wrote:
> (me again)...
> Well i try it a lot of things and finally i get a proof of concept of the final application semi functional using Qt+OpenCV+QGstreamer ...
> but.. yet.. i have a problem..
>
> Here some snippets of code
>
> main.cpp
>  int main(int argc, char *argv[]){
>    QCoreApplication a(argc, argv);
>    QGst::init(&argc,&argv);
>    FrameBuffer *buffer = new FrameBuffer(3);
>    FrameBuffer *buffer2 = new FrameBuffer(3);
>    CaptureThread *capture = new CaptureThread(buffer,0);
>    ProcessingThread *processing = new ProcessingThread(buffer,buffer2,capture->getSourceWidth(),capture->getSourceHeight());
>    StreamThread     *stream     = new StreamThread(buffer2);
>    QCoreApplication::connect(processing,SIGNAL(newframe()),stream,SLOT(update()),Qt::QueuedConnection);
>    capture->start();
>    processing->start();
>    stream->start();
>    return a.exec();;
> }
>
> buffer and buffer2 are Circular Buffers that hold cv::Mat (OpenCV Images).
> buffer hold cv::Mat obtained with CaptureThread. ProcessingThread pull images from here.
> ProcessingThread push images into buffer2 and StreamThread pull images from here.
>
> The cycle works good...
> now the StreamThread
>
>  StreamThread::StreamThread(FrameBuffer *b, QObject *parent): ImageBuffer(b),QThread(parent){
>    QString pipe_str = QString("appsrc name=\"appsrc\" caps=\"video/x-raw-rgb,width=320,height=240\" ! queue ! videoparse format=14 width=%1 height=%1 ! videorate "
>                               " ! videoscale ! video/x-raw-rgb,width=320,height=240 "
>                               " ! queue ! ffmpegcolorspace ! jpegenc  ! queue  "
>                               " ! tcpserversink port=5000  ").arg(WIDTH,HEIGHT);

Why on earth are you using so many threads here. Each queue starts a
new thread; additionally, appsrc starts a new thread too and maintains
an internal queue (making the second queue right after it completely
useless).

>    this->pipeline = QGst::Parse::launch(pipe_str).dynamicCast<QGst::Pipeline>();
>    QGlib::connect(this->pipeline->bus(),"message",this,&StreamThread::onBusMessage);
>    this->pipeline->bus()->addSignalWatch();

This is where the QTimer starts, inside the signal watch. You can see
the code here [1] if you want. This timer expires every once in a
while and polls the GstBus for new messages. It's purpose is to
deliver messages to the main thread. I'm not sure why it shows that
error message though on the output... I'll check it, it doesn't seem
normal.

>    this->pipeline->setState(QGst::StatePlaying);
>    this->m_src.setElement(this->pipeline->getElementByName("appsrc"));
> }
> void StreamThread::onBusMessage(const QGst::MessagePtr &message){
>    switch (message->type()) {
>    case QGst::MessageEos:
>        quit();
>        break;
>    case QGst::MessageError:
>        qCritical() << message.staticCast<QGst::ErrorMessage>()->error();
>        break;
>    default:
>        break;
>    }
> }
> void StreamThread::run(){
>    this->exec();
> }

I wonder if you understand that this event loop here is completely
useless. The update() slot will NEVER be called in this thread,
because this QThread object, being a QObject itself, lives in the main
thread and all of its slots are called in the context of the main
thread. So, the whole "StreamThread" could actually be a QObject
instead of a QThread and still do the same job. Let me add that since
appsrc itself handles queuing, you don't really need this extra thread
here.

> //This SLOT fires up when ProcessingThread calls the newImage() SIGNAL ..
> // this signal ocurrs when a new processing image was push into the buffer
> void StreamThread::update(){
>    if(!this->m_src.element().isNull()){
>        this->pipeline->setState(QGst::StatePlaying);

Why are you setting the pipeline *again* to playing state?

>        QGst::BufferPtr buffer = QGst::Buffer::create(WIDTH*HEIGHT*3*sizeof(quint8*));
>        quint8 *data = buffer->data();
>        Mat frame = this->ImageBuffer->getFrame();
>        data      = (quint8*)frame.data;

I hope you realize that this statement is void, it copies no actual
data. You need to copy the data with memcpy() instead.

>        QGst::FlowReturn ret = this->m_src.pushBuffer(buffer);

...and effectively here you are pushing an empty buffer. No wonder why
the other end doesn't receive anything.

>        if(ret!=QGst::FlowOk){
>            qDebug()<<"Error pushing data";
>        }
>    }
> }
> StreamThread::~StreamThread(){
>    this->pipeline->setState(QGst::StateNull);
> }
>
> When i run this app i don't get errors, just this warning (this doesn't show if i don't execute StreamThread)
> QObject::startTimer: QTimer can only be used with threads started with QThread
> So i suppose that QGstreamer use QTimer in some hidden part.
>
> And... if i try to see the stream using
> gst-launch tcpclientsrc host=127.0.0.1 port=5000 !  jpegdec ! queue ! video/x-raw-rgb,width=320,height=240 ! ffmpegcolorspace ! queue ! glimagesink sync=false
>
> I get:
> Estableciendo el conducto a PAUSA ?
> El conducto est? PREPAR?NDOSE ?
>
> But the pipeline never goes to PLAYING and nothing happen (i can't see the result)...
>
> Any idea??
>
> Thanks in advance

Regards,
George

[1]. http://gstreamer.freedesktop.org/data/doc/gstreamer/head/qt-gstreamer/html/bus_8cpp_source.html