Timestamping frame buffers correctly

[Nicolas Dufresne]

Le samedi 08 février 2020 à 16:23 -0600, jonmccormack a écrit :
> Hello all,
> I've been trying to learn how to use Gstreamer for the past few weeks, using
> the appsrc element in a c++ application stream video to an RTMP server.
> 
> I have raw frame data coming in from another part of my application, which
> has the timestamp the frame was created in milliseconds since the unix
> epoch.
> 
> Being a total beginner with regards to working with video streaming, it
> looks like timestamping the frame buffers that are piped into the gstreamer
> pipeline is essential, especially when creating a networks stream. It looks
> like the buffers are timestamped in nanoseconds since the moment the
> gstreamer pipeline was created.

Not exacty, for live sources (with a GstSegment of 0 to infinity),
which is the case with appsrc, the timestamp should match the running
time this buffer has been created. As you want to use this to tell the
pipeline when to render, you will have to set how much time it took
before these buffers actually reach GStreamer and configure this as the
min-latency.

> 
> I thought I would be able to save a record of when the pipeline was created,
> then use the following formula to timestamp the frames appropriately:
> 
> (frame timestamp (ms) - pipeline start time (ms)) * 1000000

As you have two clocks, you need to translate from one clock to
another. So you'll need to get the block from GStreamer. And do
something in these lines (assuming everything is nanosecond for
simplicity):

  unix_now = (GstCLockTime)g_get_real_time() * (GstClockTime)1000
  gst_now = gst_clock_get_time();
  
  /* We need find gst_ts in: unix_now - unix_ts == gst_now - gst_ts */
  gst_ts = gst_now - (unix_now - unix_ts)

So now we have a clock timestamp, the buffer PTS has to be running
time, so that will be:

  buf_ts = gst_ts - gst_element_get_base_time(element);

This isn't perfect, prone to scheduling delays, but (unless I made a
mistake) should give you a proper timestamp. If you don't know the
latency, you can always estimate it with:

  latency = (unix_now - unix_ts)

Like on first buffer. But it could also be very variable, in which case
you'd need something a little smarter, and maybe more observation.
There might be a way with a GstClock implementation, and slaving, not
sure if it's simpler or worst though.

> 
> However, the avenc_flv element fails to encode the buffers as the pts
> (presentation time stamp (I believe)) is invalid. The output suggests the
> timestamp of the frame buffer is invalid (pts invalid (0) <= last (0)).
> 
> My pipeline is defined as follows:
> 
> 
> And my function to pipe data into the pipeline is as follows:
> 
> 
> Could anyone offer some advice for fixing my problem?
> 
> Thanks in advance 
> Jon
> 
> P.S. I appriciate there may be some fine tuning required to improve
> streaming performance etc, but I was hoping to get a basic RTMP stream
> working first before I began tweaking. When outputting to the autovideosink
> sink element, the stream works fine, though I do believe that is more
> forgiving regards frame timestamps :)
> 
> 
> 
> 
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel