ZeroCopy + AppSink + VAAPI + Qt
Timtchenko, Michael
Michael.Timtchenko at agcocorp.com
Tue Feb 4 14:00:04 UTC 2020
Hi all,
first of all: thx to Nicolas Dufresne your tips helped me a lot :)
I rewrote my pipe, in order to use DMABuf:
"udpsrc port=55555 caps=\"application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96\" "
"! rtph264depay "
"! vaapih264dec "
"! vaapipostproc name=postproc "
"! video/x-raw(memory:DMABuf) "
"! appsink name=sink emit-signals=true sync=false max-buffers=1 drop=true"
Now i'm facing little problems with pulling the image as well as transforming it to a valid opengl texture :-/
Even now feel i feel more confident, this streaming thing still remains hard stuff.
My goal is still to get a zero-copy video stream within my QtScreneGraph-Application.
In my pull-Method i'm doing the following calls, but i'm not sure, if they are right.
pull()
{
GstSample* sample = gst_app_sink_pull_sample( m_appSink );
GstBuffer* buffer = gst_sample_get_buffer( sample );
GstCaps* caps = gst_sample_get_caps( sample );
GstMapInfo info;
gst_buffer_map( buffer, &info, GST_MAP_READ );
auto mem = gst_buffer_peek_memory( buffer, 0 );
auto fd = gst_dmabuf_memory_get_fd( mem );
// Not sure whether this is right
auto display = gst_gl_display_new();
auto context = gst_gl_context_new( display );
GstVideoInfo vInfo;
gst_video_info_from_caps( &vInfo, caps );
// What are the last two parameters used for?
// In the doc they say:
// gint plane, gsize offset
// but what plane and what offset is required here?
// Unfortunately the return value is still zero
auto image = gst_egl_image_from_dmabuf( context, fd, &vInfo, 0, 0 );
...
}
What context is supposed to get passed into gst_egl_image_from_dmabuf(xxx). Is it the target context, where i want to use the fd, or is it a different one?
But my biggest problem is, that i don't have a glue, how i can retrieve a texture from the given eglimage resp. its fd. I cannot find any sources on the web, related to EglImage texture-conversions or anything like that.
Could anyone please provide some help... i would really appreciate it.
Kind regards,
Michael
-----Ursprüngliche Nachricht-----
Von: gstreamer-devel [mailto:gstreamer-devel-bounces at lists.freedesktop.org] Im Auftrag von Nicolas Dufresne
Gesendet: Donnerstag, 30. Januar 2020 16:10
An: Discussion of the development of and with GStreamer <gstreamer-devel at lists.freedesktop.org>
Betreff: [External] Re: ZeroCopy + AppSink + VAAPI + Qt
Le jeudi 30 janvier 2020 à 09:45 +0000, Timtchenko, Michael a écrit :
> Hi folks,
>
> i'm trying to implement an efficient, zero copy image handling on a
> EGL-System without a window manager. Unfortunately this seems to be a
> quite hard job. :-/
>
> My application uses the following pipeline:
>
> "udpsrc port=55555 caps=\"application/x-rtp, media=(string)video,
> clock- rate=(int)90000, encoding-name=(string)H264, payload=(int)96\" "
> "! rtph264depay "
> "! vaapih264dec "
> "! vaapipostproc name=postproc "
> "! video/x-raw, format=RGBA "
> "! appsink name=sink emit-
> signals=true sync=false max-buffers=1 drop=true"
In this context, you are asking your GPU to upload to system memory. As the GPU does not operate in LINEAR pixel formats, this requires a copy (detiling, decompression, etc.). If you goals is to manually import your decoded image to GL, I would suggest requesting DMAbuf, VASurface, or the deprecated GstVideoGLTextureUploadMeta.
video/x-raw\(memory:DMABuf\)
video/x-raw\(memory:VASurface\)
video/x-raw\(meta:GstVideoGLTextureUploadMeta\)
And then use the appropriate API to zero-copy import that to EGL. With the preferred method (DMABuf), you need to get the GstMemory and get the DMABuf FD using gst_dmabuf_memory_get_fd(). The rest is EGL calls, so left to you.
>
> As you can see, i'm using vaapi elements as well as the appsink. I'm
> getting notified by signal, when a new frame is received.
> Afterwards i'm pulling the frame by the following code:
>
> #####
>
> GstSample* sample = gst_app_sink_pull_sample( m_data->appSink );
> GstBuffer* buffer = gst_sample_get_buffer( sample );
>
> GstMapInfo* map = new GstMapInfo();
> gst_buffer_map( buffer, map, GST_MAP_READ );
>
> and access the data by
>
> map->data()
>
> ####
>
> This is the status quo, but my goal is to put/bind the data into a
> texture and render it within the qt screnegraph without any additional
> copy actions. To achieve this, I need to get a already rendered
> texture or a direct access to the framebuffer object. Currently i'm
> uploading the data with the OpenGL-function glTexImage2D which isn't
> very efficient at all.
>
> In (1) they said, that this can be done by:
>
> [...Using memory:VASurface you can map it as a normal gst video frame:
> gst_video_frame_map ...]
>
> Unfortunately i don't have a clue, how i can access a texture or a
> framebufferobject from the VASurface. Is that even possible?
>
> Is there someone who can share his knowledge? Any help is welcome! :)
>
> Thanks in advance,
> Michael
>
> (1)
> http://gstreamer-devel.966125.n4.nabble.com/How-to-use-VAAPI-appsink-w
> ith-zero-copy-mmap-td4688670.html
>
>
> Bitte beachten / Please note!
>
> **********************************************************************
> ********
> *
>
> AGCO GmbH
> Sitz der AGCO GmbH: Johann-Georg-Fendt-Str.4, 87616 Marktoberdorf,
> Germany Registergericht Amtsgericht Kempten HRB 10327
> Geschäftsführer: Christoph Groeblinghoff, Ingrid Bussjaeger-Martin, Dr.
> Heribert Reiter, Ekkehart Glaeser
> Vorsitzender des Aufsichtsrates: Torsten Dehner
>
> **********************************************************************
> ********
> *
>
> Diese E-Mail ist nur für den Empfänger bestimmt, an den es gerichtet
> ist und kann vertrauliches bzw. unter das Berufsgeheimnis fallendes
> Material enthalten. Jegliche darin enthaltene Ansicht oder
> Meinungsäußerung ist die des Autors und stellt nicht notwendigerweise
> die Ansicht oder Meinung von AGCO dar. Sind Sie nicht der Empfänger,
> so haben Sie diese E-Mail irrtümlich erhalten und jegliche Verwendung,
> Veröffentlichung, Weiterleitung, Abschrift oder jeglicher Druck dieser
> E-Mail ist strengstens untersagt. Weder AGCO noch der Absender
> übernehmen die Haftung für Viren; es obliegt Ihrer Verantwortung, die
> E-Mail und deren angehängte Dateien (sofern vorhanden) auf Viren zu
> überprüfen.
>
> **********************************************************************
> ********
> *
>
> This email is intended solely for the use of the individual to whom it
> is addressed and may contain confidential and/or privileged material.
> Any views or opinions presented are solely those of the author and do
> not necessarily represent those of AGCO. If you are not the intended
> recipient, be advised that you have received this email in error and
> that any use, dissemination, forwarding, printing or copying of this
> email is strictly prohibited. Neither AGCO nor the sender accepts any
> responsibility for viruses and it is your responsibility to scan and
> virus check the email and its
> attachment(s) (if any).
>
> **********************************************************************
> ********
> *
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
This email is intended solely for the use of the individual to whom it is addressed and may contain confidential and/or privileged material. Any views or opinions presented are solely those of the author and do not necessarily represent those of AGCO. If you are not the intended recipient, be advised that you have received this email in error and that any use, dissemination, forwarding, printing or copying of this email is strictly prohibited. Neither AGCO nor the sender accepts any responsibility for viruses and it is your responsibility to scan and virus check the e-mail and its attachment(s) (if any).
More information about the gstreamer-devel
mailing list