gst-plugins-good: multipartmux extensions

[Robin Haberkorn]

Hello Sebastian,

On Sat, 12 Mar 2016 11:41:53 +0200
Sebastian Dröge <sebastian at centricular.com> wrote:

> On Fr, 2016-03-11 at 20:34 +0100, Robin Haberkorn wrote:
> > 
> > Furthermore I implemented support for adding custom HTTP header
> > fields generated from metadata associated with GstBuffers. Ie. I
> > have defined a GstMeta implementation that allows you to add some
> > GstStructure to a GstBuffer as meta data. If the name of the
> > structure is "http-headers" (could be easily changed of course),
> > one HTTP header is added per GstStructure field with a
> > gst_value_serialize'd value.
> 
> There's a custom event for similar purposes in souphttpsrc. Maybe
> those two things can be merged somehow, the advantage of the event is
> that it's independent of buffers (and IIRC it was a sticky event).
> 
Thanks for pointing that out. It's also entirely
undocumented :-)
The fact that events are independent of buffers is exactly why I did
not use them, though. I use meta data to convey information about the
contents of a buffer (always a video frame in my case). Take for
instance the result of a computer vision analysis on a video stream.

I think that sticky events would be OK for adding custom headers in
multipartmux. But what if you want to process the meta data in an
application using GstAppSink? AFAIK if I had events instead of GstMeta,
I could only get the event from the pipeline bus and I'd loose the
association with individual buffers.

Perhaps a good compromise would be to support both a custom GstMeta
type and souphttpsrc's http_headers event.
What do you think?

> > My question is whether you are willing and interested to merge all
> > of those changes back into the upstream project.
> > If yes, it makes sense to keep my modifications as minimal as
> > possible; keeping the type and short name of the modified element
> > etc.
> 
> That certainly sounds interesting, as long as it is done in a
> backwards compatible way :)
> 
> Note that multipart is not HTTP specific though, so let's try to keep
> things somehow generic.
> 
Indeed it is not necessarily restricted to HTTP. Somebody could stream
out a video using E-Mail. Play By Email ;-). But seriously, multipart 
messages are defined in RFC1341. For body parts it refers to RFC822
but it also specifically mentions that additional header fields can
occur. If they don't start with "Content-" they are to be ignored in
general. It also says: '"X-" fields may be created for experimental or
private purposes.' So adding timestamp and dynamic headers should be
allowed by the specs.
Furthermore, my changes would be backwards-compatible: X-Timestamp
would be disabled by default -- you will have to enable it using a new
property. The additional meta-data headers are only added if the
corresponding meta-data exists for the buffer. But it could also be
made entirely optional, e.g. by making the GstStructure name
configurable in multipartmux.

> 
> > My extensions add additional multipart HTTP headers. Firstly, an
> > X-Timestamp header is added with the buffer timestamp. This is
> > useful in our case if the pipeline clock is a real-time system
> > clock, so we can put individual frames recorded using a live source
> > and muxed using multipartmux into chronological relation with other
> > frames (possibly recorded by independant live sources).
> 
> Independent of the other discussion, keep in mind that such header
> requires you to somehow synchronize the clocks of the senders and
> receivers and compensate for different clock rates of their own
> clocks. At least if it is to be used for synchronization of the media.

I'm using PTP to synchronize the real-time system clocks. That should do
the job.
Currently I'm using linuxptp to synchronize the system clock and I'm
enforcing the realtime system clock on my pipeline.

I'm aware that you have implemented a dedicated GstPtpClock at
Centricular. However I don't see the benefit in using it in favour
linuxptp+system clock. Perhaps you can enlighten me.

> You could add another header that gives information about which clock
> is actually used here though (see e.g. RFC7273 which has that for
> RTP).
> 

You mean something like the ts-refclk? Could be useful in the long run,
yes. Esp. to debug synchronization issues in more complex setups I
guess. The information that PTP is used as the clock source is not
really useful - in my case at least - since that's a defined part of the
product.
But how to report the PTP server in the context of Gstreamer?
GstPtpClock could probably report that. That wouldn't be
clock-agnostic though, as GStreamer does not seem to support
generic GstClock identification.
When using some extern way to synchronize the system clock, I don't
think any automatic clock identification will work though. So why not
just make that user/application settable? There could be yet another
multipartmux property for adding arbitrary header fields to each
multipart segment not derived from the GstBuffer. Something like
souphttpsrc's "extra-headers". The application may then manage
header fields identifying the clock source.
What do you think?

Best regards,
Robin