[Bug 685282] New: Figure out how to make DVD SPU work with hardware decoders and subpicture overlays

Tue Oct 2 06:37:03 PDT 2012

https://bugzilla.gnome.org/show_bug.cgi?id=685282
  GStreamer | gst-plugins-bad | git

           Summary: Figure out how to make DVD SPU work with hardware
                    decoders and subpicture overlays
    Classification: Platform
           Product: GStreamer
           Version: git
        OS/Version: Linux
            Status: NEW
          Severity: enhancement
          Priority: Normal
         Component: gst-plugins-bad
        AssignedTo: gstreamer-bugs at lists.freedesktop.org
        ReportedBy: thaytan at mad.scientist.com
         QAContact: gstreamer-bugs at lists.freedesktop.org
     GNOME version: ---

Getting the DVD SPU to paint generically is a requirement for allowing the DVD
elements to plug / output hardware decoder caps.

Here's a conversation we had about it on IRC:

Sep 26 01:37:04 *       thaytan wonders how SPU works
Sep 26 01:37:19 <bilboed>       thaytan, with vdpau ?
Sep 26 01:37:24 <thaytan>       *nod*
Sep 26 01:37:35 <thaytan>       how it should work, that is
Sep 26 01:37:36 <bilboed>       they have also VdpBitmapSurface
Sep 26 01:38:01 <bilboed>       so VdpVideoSurface => YUV stuff,
VdpBitmapSurface => RGB stuff
Sep 26 01:38:11 <bilboed>       VdpOutputSurface => output of the compositor
for display
Sep 26 01:38:40 <bilboed>       you can create VdpBitmapSurface and map(write)
stuff on it
Sep 26 01:39:01 <bilboed>       then you give it to the compositor with
coordinates (and whatever else) and that's basically it
Sep 26 01:39:09 <thaytan>       hmmm
Sep 26 01:39:26 <bilboed>       so as a result ... I'm also gonna have to
figure out how to solve hw-compositing
Sep 26 01:39:28 <thaytan>       not sure I see how that fits into a
GStreamer/rsndvdbin context
Sep 26 01:39:31 <bilboed>       in a generic fashion that is
Sep 26 01:39:55 <bilboed>       thaytan,  was thinking you could slap
GstOverlayMeta (or sth like that) with attached GstBuffers
Sep 26 01:40:06 <bilboed>       thaytan, maybe videomixer or some generic
element could do that
Sep 26 01:40:09 <thaytan>       I guess it's a vdpspu element with video and
subpicture inputs as currently with dvdspu
Sep 26 01:40:25 <thaytan>       except the video pad accepts vdp output surface
caps
Sep 26 01:40:38 <bilboed>       I'd prefer to avoid creating
yet-another-custom-element
Sep 26 01:40:45 <thaytan>       but, I don't know what it outputs
Sep 26 01:41:12 <thaytan>       bilboed: I don't know how to build it
generically
Sep 26 01:41:48 <thaytan>       the dvdspu element uses the video stream
passing through to a) paint onto b) uses the timestamps to crank the SPU state
machine
Sep 26 01:42:44 <bilboed>       what do you need more apart from "put this RGB
bitmap at these coordinates for this timestamp and this duration"
Sep 26 01:43:48 <thaytan>       it needs the timestamps and segment info on the
video stream so it knows which pixels to generate
Sep 26 01:44:04 <thaytan>       it's more "here's a video frame, what's the
overlay?"
Sep 26 01:44:13 <thaytan>       also, dvdspu works in YUV too
Sep 26 01:44:13 <bilboed>       sure, but it doesn't care about the *content*
of that frame
Sep 26 01:44:28 <thaytan>       bilboed: not if it's not doing the compositing,
no
Sep 26 01:44:34 <bilboed>       right
Sep 26 01:45:11 <bilboed>       so it could see the stream go through,
watch/collect segment/timestamps and decide what subpicture to attach to it
(without *actually* doing any processing and letting downstream handle that)
Sep 26 01:45:13 <thaytan>       but the model is still to pass the video buffer
stream through the spu element so it can see the timing info it needs, right?
Sep 26 01:45:22 <thaytan>       oh, of course
Sep 26 01:45:28 <thaytan>       that's what I was suggesting, I guess I wasn't
clear
Sep 26 01:45:35 <__tim> thaytan, dvdspu should support
GstVideoOverlayComposition imho
Sep 26 01:45:38 <bilboed>       I'm not *that* familiar with SPU
Sep 26 01:45:42 <bilboed>       also, what __tim said :)
Sep 26 01:45:54 <bilboed>       like that I don't have to solve it in 500
different elements
Sep 26 01:45:56 <thaytan>       ok, I guess I'll have to look at the
GstVideoOverlayComposition API
Sep 26 01:46:54 *       bilboed is not looking forward "at all" to fixing this
for cluster-gst
Sep 26 01:47:12 <__tim> it's very dumb, you can provide one or more ARGB or
AYUV rectangles and either use helper API to put them onto the raw video, or
attach them to the buffer; the sink or whatever can then take over the
overlaying using that
Sep 26 01:47:40 <__tim> and it will do a bunch of conversions for you and cache
them if the sink or whatever does the overlaying doesn't support what you
supplied
Sep 26 01:48:54 <thaytan>       well, that sounds feasible - although less
efficient than dvdspu painting natively if the composite is in software
Sep 26 01:49:28 <thaytan>       maybe it can be extended to add RLE AYUV
rectangles as a format though?
Sep 26 01:49:44 <__tim> thaytan, how sure are you of that? because basically
you have to parse the RLE data for every single frame, right? is that really so
much faster than blitting some ready-made rectangle using orc?
Sep 26 01:50:04 <thaytan>       dvdspu gets to skip a lot of transparent pixels
Sep 26 01:52:21 <__tim> yeah, but it's if else and loops etc. You might be
right, I'm just curious how much difference it actually makes. Also, you don't
have to use the API to blit your pixels, you can still do that as you do now
and only attach the AYUV rectangle if downstream supports that
Sep 26 01:54:13 <__tim> it's just convenient because you only have one code
path
Sep 26 01:55:01 <thaytan>       it sounds like a structural improvement

-- 
Configure bugmail: https://bugzilla.gnome.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
You are the assignee for the bug.