[gst-devel] GStreamer and Closed-Captioning support

[David I. Lehn]

* Stefan Kost <ensonic at hora-obscura.de> [2008-01-08T04:10:44-0500]:
> Quoting Stefan Kost <ensonic at hora-obscura.de>:
> > Quoting Bryen Yunashko <byunashko at gmail.com>:
> >
> >> Can anyone send me information on whether gstreamer supports
> >> closed-captioning with gstreamer?  Particularly, I'm trying to find it in
> >> Totem.
> > Could you elaborate a bit what that is? Is it reading out subtitles?
> > Any pointers? I want to prepare a list of taks for the next google
> > summer-of-code application. This could be a potential subproject.
> 
> Replying to myself:
> http://en.wikipedia.org/wiki/Closed_captioning
> 
> Bryen, do you have any info about how this differs from subtitles? Do  
> you have some sample content that has closed captioning tracks that  
> you could give to the gstreamer developers?
> 

When I was active here back in the day (years ago) I was playing with CC
code in GStreamer.  I think some of my code may have been lost to time
though.

At one point I had GStreamer decoding the CC data from DVDs.  It wasn't
doing full positional on-screen rendering but just dumping the raw text
to the console.  At the time I was quite thrilled to get that far but
moved on to other projects.

To get it working I made a specific Bin source on the mpeg demuxer (I
think?) to output the "user_data" packets.  On a DVD that is how they
store the VBI data.  I then stole the VBI code from the very excellent
tvtime project to do the hard work.  I think it's still in the "bad"
plugins repo in the vbidec plugin.  Then you just route the user_data to
vbidec and the output of vbidec to a console.  (Or close to that...
might have been stripping off control codes to just print the text.)  It
worked just fine but was mostly just a testing hack for GStreamer, DVD
code, Python bindings, and other stuff... all at once.  Would be easier
today.

To make this code useful there's lots of work to be done.

- Update vbidec to latest plugin standards and APIs.  I'm sure it's
  quite bitrotten by now.
- Look at status of the "user_data" mpeg fields.  I have no idea what it
  is these days.  I think it was ripped out at some point due to the
  core not being able to handle multiple bin sources at once properly or
  something.  Might be easier today or already done.  This is the
  easiest way to test CC data.  It's a correct VBI bitstream on any DVD
  with CC data.  Live video signals would work too but might have errors
  in the VBI data.  Though it's supposed to handle some level of errors.
- Try pushing V4L streams through vbidec.  I don't recall if I ever
  tried this.  I also can't remember if you have to strip out the VBI
  info or if the vbidec code can find that for you.
- Take the decoded VBI info and do something more useful with it.  There
  is data in there for display timing, colors, symbols, fonts,
  positional info... etc etc.  TVTime also supported other type of info.
  For instance the tv channel/program info (XDS) and the menu system
  more popular in .eu than .us.
- Create something to render the CC data.  (Re-use subtitle related
  work)
- Create something to composite that rendered data stream on top of a
  video stream properly.  (Re-use subtitle related work)
- Hook it all up, integrate into DVD and V4L apps and add some controls.
- Check with the tvtime.sourceforge.net project to see if they have any
  newer code.  Also you might need to rip more code to support more than
  just CC data.  TVTime would be _excellent_ reference code for the
  non-GStreamer-specific parts of this task.

There may be an issue with the vbidec code being in GStreamer at all.
It comes from a GPL project.  I'm not sure what the licensing issues are
these days with that.  Might need to convert bits of tvtime into an
external library and keep GStreamer specific parts in gst plugins.

This would all make an interesting and useful project.  With example
code like TVTime out there it's probably not extremely hard either.

I had always tried to convince vektor (Billy Biggs) from tvtime fame to
just use GStreamer as a base for tvtime but that never happened.  They
were focused on making the most elite tv app ever and succeeded.  Might
be possible at this point to port over pieces of it beyond just the CC
related parts.  There are some issues with GStreamers frame based
algorithms (I assume it's still like that?) and tvtimes scan-line
algorithms.  Fun stuff.  Good luck.

-dave