gst-plugins-good: multipartmux extensions

[Robin Haberkorn]

Hello Sebastian,

On Sun, 13 Mar 2016 11:28:31 +0200
Sebastian Dröge <sebastian at centricular.com> wrote:

> ...
> 
> My idea was that both the event and the meta could share the
> representation at least, and that for non-per-buffer headers you could
> use the event and for per-buffer headers the meta. I.e. having the
> event as the default values and let the meta override them.
> 
Yes, makes sense.

> ...
> 
> linuxptp+system clock requires you to synchronize your system clock to
> one specific PTP clock, and it requires root permissions to actually
> do so. With the GStreamer PTP clock you can synchronize your media to
> different PTP/NTP/etc servers without affecting your system clock time
> and without requiring root permissions.
> 
Both of these advantages are irrelevant to me as I'm developing an
embedded product. My biggest concern on using GstPtpClock is however
that it is a recent feature only implemented in 2015. I'd have to update
my on-board GStreamer - or backport GstPtpClock - which may be a lot
of work. It's not always pleasant to upgrade a large project like
GStreamer in an embedded system as all components must fit together.
(Of course I will have similar problems with my multipartmux changes,
but it's relatively isolated.) So for the sake of simplicity I'm staying
with linuxptp at least for the time being.

btw. I had compatibility problems with OpenCV and thus patched
gst-plugins-bad to compile with OpenCV 3. I should probably post that
here. Unfortunately, the code base of gst-plugins-bad has changed so
much recently that this patch won't work anymore. At the same time
gst-plugins-bad is still not OpenCV 3 compatible. So when I upgrade
gstreamer, I will have to completely rewrite that, too. At least the
patch may then be merged upstream, finally adding support for OpenCV 3.

> ...
> > You mean something like the ts-refclk? Could be useful in the long
> > run, yes. Esp. to debug synchronization issues in more complex
> > setups I guess. The information that PTP is used as the clock
> > source is not really useful - in my case at least - since that's a
> > defined part of the product.
> 
> The main advantage here is IMHO that you have this information next to
> media instead of having to distribute it somehow else (like making it
> a requirement of your product that the system clock is synchronized to
> that specific PTP clock).
> 
My use case is probably much different than what you have in mind.
Clients receiving the video data will not have to sync against the same
clock. But I acknowledge that it's useful elsewhere.

> ...
> 
> See https://bugzilla.gnome.org/show_bug.cgi?id=760005 and
> https://bugzilla.gnome.org/show_bug.cgi?id=762259
> 
> Yes, it actually checks for those two clocks and handles them
> specifically right now. The number of possible clocks here seems
> rather limited, so I didn't want to overengineer it :)
> 
Alright, I will add similar code for NTP and PTP clocks.

> ...
> > When using some extern way to synchronize the system clock, I don't
> > think any automatic clock identification will work though. So why
> > not just make that user/application settable? There could be yet
> > another multipartmux property for adding arbitrary header fields to
> > each multipart segment not derived from the GstBuffer. Something
> > like souphttpsrc's "extra-headers". The application may then manage
> > header fields identifying the clock source.
> > What do you think?
> 
> Makes sense. But the "ts-refclk" (vocals seem to be expensive) is not
> really required for getting your patches merged anyway IMHO :)
> 
What would you like to call it instead? X-Timestamp-RefClock?

If you use "extra-headers" for externally synced system clocks, you can
of course choose whatever name you like.

"extra-headers" would consequently be merged into the event and meta
data headers (after both of them).

> If that ever becomes necessary for the RFC7273 implementation in the
> RTSP server or jitterbuffer (see patches), API could be added for
> specifying that the system clock is indeed synced to one specific
> PTP/NTP/etc clock. Or it could even be API on the system clock.
> 
Mhh... I don't think that's a good idea. What if you sync your system
clock to something else, like NTP? This can hardly be mapped generically
to SystemClock properties or methods. And even if it could be by adding
some new properties, you could not ask the OS about the relevant
properties; so you would have to update the SystemClock properties from
your App anyway using whatever means necessary.
So why not just use the "extra-headers" way of delegating that task to
the application?

> 
> FWIW, note that the system clock used by default is the monotonic one
> and not the real-time one. So you will get rather arbitrary values
> there instead of the wallclock time according to the PTP clock. You
> can change that though (just be careful when the clock changes then,
> like when your system suspends and then suddenly there is a huge
> clock jump or the system clock jumps backwards or forwards because of
> linuxptp adjustments, or ...).
> 
I'm aware of that and have done that. That's also why I'd like to keep
a rather idiosyncratic X-Timestamp format: it is simply the sum of
the buffer's running time and the element's base time in nanoseconds,
resulting in an absolute timestamp with regard the pipeline clock's
origin or epoch.
If you set up a realtime system clock, that just happens to be the UNIX
epoch and the X-Timestamp happens to be in nanoseconds since the UNIX
epoch. If you use some other monotonic clock, the timestamp value will
probably be useless but at least easy to describe. That's why it makes
little sense to use something like the NTP timestamp format since it
has an explicit epoch and I can only translate the buffer's timestamp
to an NTP timestamp when I know a little bit about the pipeline's clock.
Otherwise I would have to __assume__ something like the UNIX epoch,
complicating the timestamp semantics (if the pipeline clock is NOT
based on the UNIX epoch).

Best regards,
Robin