gst-plugins-good: multipartmux extensions

Sun Mar 13 09:28:31 UTC 2016

On Sa, 2016-03-12 at 21:09 +0100, Robin Haberkorn wrote:
> Hello Sebastian,
> 
> On Sat, 12 Mar 2016 11:41:53 +0200
> Sebastian Dröge <sebastian at centricular.com> wrote:
> 
> > 
> > On Fr, 2016-03-11 at 20:34 +0100, Robin Haberkorn wrote:
> > > 
> > > 
> > > Furthermore I implemented support for adding custom HTTP header
> > > fields generated from metadata associated with GstBuffers. Ie. I
> > > have defined a GstMeta implementation that allows you to add some
> > > GstStructure to a GstBuffer as meta data. If the name of the
> > > structure is "http-headers" (could be easily changed of course),
> > > one HTTP header is added per GstStructure field with a
> > > gst_value_serialize'd value.
> > There's a custom event for similar purposes in souphttpsrc. Maybe
> > those two things can be merged somehow, the advantage of the event is
> > that it's independent of buffers (and IIRC it was a sticky event).
> > 
> Thanks for pointing that out. It's also entirely undocumented :-)
>
> The fact that events are independent of buffers is exactly why I did
> not use them, though. I use meta data to convey information about the
> contents of a buffer (always a video frame in my case). Take for
> instance the result of a computer vision analysis on a video stream.
> 
> I think that sticky events would be OK for adding custom headers in
> multipartmux. But what if you want to process the meta data in an
> application using GstAppSink? AFAIK if I had events instead of GstMeta,
> I could only get the event from the pipeline bus and I'd loose the
> association with individual buffers.
> 
> Perhaps a good compromise would be to support both a custom GstMeta
> type and souphttpsrc's http_headers event.
> What do you think?

My idea was that both the event and the meta could share the
representation at least, and that for non-per-buffer headers you could
use the event and for per-buffer headers the meta. I.e. having the
event as the default values and let the meta override them.

> > > My question is whether you are willing and interested to merge all
> > > of those changes back into the upstream project.
> > > If yes, it makes sense to keep my modifications as minimal as
> > > possible; keeping the type and short name of the modified element
> > > etc.
> > That certainly sounds interesting, as long as it is done in a
> > backwards compatible way :)
> > 
> > Note that multipart is not HTTP specific though, so let's try to keep
> > things somehow generic.
> > 
> Indeed it is not necessarily restricted to HTTP. Somebody could stream
> out a video using E-Mail. Play By Email ;-).

There's a curlsmtpsink ;)

>  But seriously, multipart 
> messages are defined in RFC1341. For body parts it refers to RFC822
> but it also specifically mentions that additional header fields can
> occur. If they don't start with "Content-" they are to be ignored in
> general. It also says: '"X-" fields may be created for experimental or
> private purposes.' So adding timestamp and dynamic headers should be
> allowed by the specs.
> Furthermore, my changes would be backwards-compatible: X-Timestamp
> would be disabled by default -- you will have to enable it using a new
> property. The additional meta-data headers are only added if the
> corresponding meta-data exists for the buffer. But it could also be
> made entirely optional, e.g. by making the GstStructure name
> configurable in multipartmux.

Makes sense

> > > My extensions add additional multipart HTTP headers. Firstly, an
> > > X-Timestamp header is added with the buffer timestamp. This is
> > > useful in our case if the pipeline clock is a real-time system
> > > clock, so we can put individual frames recorded using a live source
> > > and muxed using multipartmux into chronological relation with other
> > > frames (possibly recorded by independant live sources).
> > Independent of the other discussion, keep in mind that such header
> > requires you to somehow synchronize the clocks of the senders and
> > receivers and compensate for different clock rates of their own
> > clocks. At least if it is to be used for synchronization of the media.
> I'm using PTP to synchronize the real-time system clocks. That should do
> the job.
> Currently I'm using linuxptp to synchronize the system clock and I'm
> enforcing the realtime system clock on my pipeline.
> 
> I'm aware that you have implemented a dedicated GstPtpClock at
> Centricular. However I don't see the benefit in using it in favour
> linuxptp+system clock. Perhaps you can enlighten me.

linuxptp+system clock requires you to synchronize your system clock to
one specific PTP clock, and it requires root permissions to actually do
so. With the GStreamer PTP clock you can synchronize your media to
different PTP/NTP/etc servers without affecting your system clock time
and without requiring root permissions.

> > You could add another header that gives information about which clock
> > is actually used here though (see e.g. RFC7273 which has that for
> > RTP).
> > 
> You mean something like the ts-refclk? Could be useful in the long run,
> yes. Esp. to debug synchronization issues in more complex setups I
> guess. The information that PTP is used as the clock source is not
> really useful - in my case at least - since that's a defined part of the
> product.

The main advantage here is IMHO that you have this information next to
media instead of having to distribute it somehow else (like making it a
requirement of your product that the system clock is synchronized to
that specific PTP clock).

> But how to report the PTP server in the context of Gstreamer?
> GstPtpClock could probably report that. That wouldn't be
> clock-agnostic though, as GStreamer does not seem to support
> generic GstClock identification.

See https://bugzilla.gnome.org/show_bug.cgi?id=760005 and
https://bugzilla.gnome.org/show_bug.cgi?id=762259

Yes, it actually checks for those two clocks and handles them
specifically right now. The number of possible clocks here seems rather
limited, so I didn't want to overengineer it :)

> When using some extern way to synchronize the system clock, I don't
> think any automatic clock identification will work though. So why not
> just make that user/application settable? There could be yet another
> multipartmux property for adding arbitrary header fields to each
> multipart segment not derived from the GstBuffer. Something like
> souphttpsrc's "extra-headers". The application may then manage
> header fields identifying the clock source.
> What do you think?

Makes sense. But the "ts-refclk" (vocals seem to be expensive) is not
really required for getting your patches merged anyway IMHO :)

If that ever becomes necessary for the RFC7273 implementation in the
RTSP server or jitterbuffer (see patches), API could be added for
specifying that the system clock is indeed synced to one specific
PTP/NTP/etc clock. Or it could even be API on the system clock.

FWIW, note that the system clock used by default is the monotonic one
and not the real-time one. So you will get rather arbitrary values
there instead of the wallclock time according to the PTP clock. You can
change that though (just be careful when the clock changes then, like
when your system suspends and then suddenly there is a huge clock jump
or the system clock jumps backwards or forwards because of linuxptp
adjustments, or ...).

-- 
Sebastian Dröge, Centricular Ltd · http://www.centricular.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: This is a digitally signed message part
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20160313/9660de71/attachment-0001.sig>