[Bug 766790] v4l2src generates wrong timestamps for encoded video.

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=766790

--- Comment #4 from Peter Maersk-Moller <pmaersk at gmail.com> ---
Hi Nicolas.

> (In reply to Nicolas Dufresne (stormer) from comment #3)
> I just notice that your thread is about 1.4.5 on Beagle Bone. Have you
> tested recent kernel and/or recent GStreamer ?

I thought I had tested the timestamps on my laptop (x86) with GStreamer 1.8.1.
But apparently not ... or the C920 I have here has a better firmware. I will
within some days test on various platforms and with a selection of cameras.
The Beaglebone has an older kernel and the GStreamer 1.4.5. I will try to
update both, however this is a process that will take some time.

> I have taken out my C920, and
> I cannot reproduce your result (GStreamer master and kernel 4.4). As
> expected, on older kernel I expect that the bad-timestamp code triggers, and
> timestamp endup the time when v4l2src manage to dequeue the frame. This
> might be a little jitterry, but it's the best interpolation you can do.
> Adding a queue after v4l2src usually remove the jitter. On newer kernel, the
> HW timestamp is ignored, hence the timestamp is the time capture right after
> the IRQ is triggered (which is accurate, minus the latency of course). This
> is live sources, so the moment the frame is capture, is the timestamp. If
> your camera is jittery by design, you should have jittery timestamp.

The timestamps I have measured can be due to older kernel and/or older
GStreamer. However this could, at least theoretically be due to effects more
visible when running on a platform as slow (compared to a x86 laptop) as the
BeagleBone. These effect, in theory could be caused in kernel as well in
GStreamer (theoretically, with the little knowledge I yet have on the
timestamping code of v4l2src). If the timestamps are due to effects caused by a
slower platform, then the issue is worth looking into. If not, then the matter
can be closed as no longer relevant.


> (In reply to Peter Maersk-Moller from comment #2)
> > Can you point me to some file/functions where you do the bad time stamp
> > detection and time stamp interpolation in v4l2src?
> https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/sys/v4l2/
> gstv4l2src.c#n721

Will look into it. Thanks. Please have patience as it will take some time and I
unfortunately can't spend work time on it, only my little spare time.

> We don't do interpolation from existing timestamp. My opinion is that the
> kernel should do that, because it gets much more accurate information.
> Though, that code is broken in UVC driver and was disabled temporarily in
> 4.2 until someone get the time to look at it. Obviously, if it turns out
> your firmware is broken, complain to Logitech.

Obviously, except the result will be obvious.

> > In my code, I read from a frame fifo at a constant frame rate and it works
> > pretty well even out time stamps from all these buggy webcams. The encoded
> > frames may have variable timed timestamps, but the frames them selves are
> > captured at an steady rate (and subsequently should be palyed at a steady
> > rate).
> 
> Not sure what your code is doing (share code if you want to keep that bug
> open without providing solutions). I do have deployed code using the current
> v4l2src (and also uvch264src) and it worked fine.

The the code is freely available in Snowmix feeds, but is as code not relevant
for GStreamer, since it uses a different technique for mixing and syncing
various sources. It is just a bunch fifos supplied by Gstreamer and read/used
at a steady framerate by Snowmix. Nothing fancy or worth looking into, but
using this, I can see movements between frames appear uniform smooth when I
disregard the variations in timestamps and read at a steady framerate.

> > Anyway in uvch264src, you have the parameter 'fixed-framerate'. I would have
> > expected that, when enabled, it would try to generate equal spaced time
> > stamps.
> 
> This property will send UVC_H264_RATECONTROL_FIXED_FRM_FLG to the camera. If
> your camera ignores it, complain to your vendor. I do remember Youness (this
> element author) reporting that it only worked if you use H264 muxed into
> JPEG (anything with a preview feed).

Ok, thanks, Yes I can't see any difference whether it is enabled or not.

> > True that we should not break things. However, I also believe we should in
> > general try to correct the timestamps, when possible. It could be a
> > parameter you can enable so other things that don't need/want it can ignore
> > it.
> The only completely incorrect timestamp are those that go backward.
> GStreamer/V4L2 does not let timestamp go backward for capture since 1.6, so
> if this is the problem you see, update GStreamer, or backport the fixes.

Ok.

> > True the firmware should be fixed in all these cameras, but that is not
> > going to happen. True it should be fixed in the driver and or in V4L2
> > community. Don't really know anyone there so that may not happen either.
> > True I can compensate for it in my software, although it would be really
> > nice to have it done earlier.
> > One thought. h264parse does, as I understand try to correct some TS errors.
> > It could happen there as well. Something like 'videorate', but for a H264
> > stream although adding frames is most likely not possible and throwing away
> > frames can cause problems. Admittedly I haven't thought it through. But best
> > would be to be able to correct timestamps in v4l2src or alterntively in the
> > uvch264src using v4l2src.
> You need at least one valid timestamp (DTS or PTS) to fix the timestamp in
> h264parse. At the moment, you seem to state that we have none, so this won't
> work.

Ok, thanks. Yes the timestamps observed are incorrect, so I wouldn't have
correct timestamps to work from. However what I suggest is a mechanism like the
videorate element, but an element that would be able to adaptively generate
approximated even spaced timestamps. Right now, if I wanted to even out the
space in timestamps, I would have to decode the stream, send it through
videorate and then encode the stream again. And that is obviously not
attractive. However there is a chance that what I write here is wrong in case I
have misunderstood what module videorate can do.

> Again, smoothing timestamp can be wrong, as it may not be
> representative to the capture time. There is load of people that do cares
> about capture time accuracy.

So do I. Definitely, or at least here I care about the relative values and they
happen to be wrong - for all the reasons mentioned earlier
(kernel/driver/firmware and possible GStreamer older version).

You can choose to close the ticket. It can always be reopnened if needed. I'll
look into the issues mentioned and report back if I find anything useful.

That said, a way in GStreamer to even space between timestamps without decoding
could be a useful features. So much software and hardware have a hard time
doing timestamps right and being able to use GStreamer as a tool to fix some of
it, could be worth it. But that is just me being oppinionated.

Thanks for help, explanation and tips.

Peter

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.