Query related to Closed Captioning Support in Gstreamer

[Mathieu Duponchelle]

Hey,

On 1/7/21 6:40 PM, TusharK wrote:
> Hi,
>
> Can you please help me with details on below queries related to closed
> captioning support in gstreamer
>
> 1)	Can we stream “Closed Captioned” text separately as separate channel like
> audio and video?

You can extract closed captions with ccextractor yes.
> 2)	Which streaming protocol (RTMP/RTSP/WebRTC etc.) support streaming of
> “Closed Caption” text? Any recommendation would be really helpful

I'm not aware of any streaming protocol that allows transport of
closed captions separately, but perhaps there is one I'm not aware of.

> 3)	Came across closed caption support in H264/H265 parser and encoder, this
> is getting added as SEI Registered User Data. Can we use this feature to
> stream the Closed Captioning information. At client / player side we are
> planning to use gstreamer pipeline itself for decoding. Hope that this
> closed caption data can be extracted and rendered properly. Is this
> understanding correct?

Yes, if you inject closed captions in the H264 bitstream GStreamer can
extract it on the other end.

> 4)	If we use WebRTC for streaming (H264 + closed captioning data inserted as
> SEI data) then will browser able to decode this closed captioning data and
> render it on HTML page canvas. Is this understanding correct?

Well, you will need the decoder / parser used on the browser side to support
extracting that information, and you will need a player capable of interpreting
and displaying it, but technically yes, the browser will have the information
available.

> 5)	It would be great if you could share the example gstreamer pipelines
> which demonstrates streaming and playback of closed captioned video? I gone
> through some gstreamer forum posts but could not find the reference pipeline
> along with the stream to be used.

We should probably add some index page to our documentation that
sums up our closed caption capabilities yes, for now I would advise
reading the documentation of the various closedcaption elements
at <https://gstreamer.freedesktop.org/documentation/closedcaption/index.html?gi-language=c#plugin-closedcaption>

You should also know that we have a bunch more closed caption
elements in our rust plugins, but those don't have documentation
exposed, we haven't figured out how we want to set up our infrastructure
for that yet. Your best option will be to look at the sources:

https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/tree/master/video/closedcaption/src

Apart from that, it would be nice to improve autoplugging for CC,
eg cea608overlay should be autopluggable when playbin detects
a 608 stream, this will require some effort though :)

Cheers,

-- 
Mathieu Duponchelle · https://www.centricular.com <https://www.centricular.com>