Clock skewing algorithm

[Tvrtko Ursulin]

On Tuesday 11 Oct 2011 17:10:05 Wim Taymans wrote:
> On 10/11/2011 03:15 PM, Tvrtko Ursulin wrote:
> > Hi Wim, all,
> > 
> > I was just talking with Jan Schmidt on #gstreamer about a curious clock
> > skewing observation you might sched some light at.
> > 
> > Essentially I am debugging AV sync issues when streaming video from
> > v4l2src and audio from alsasrc. To narrow it down I went on to test
> > purely audio pipeline in the form of "alsasrc !
> > audio/x-raw-int,rate=48000 ! queue min- threshold-time=2000000000 !
> > alsasink".
> 
> This pipeline is rather impossible to run, the queue will always overrun
> and block the source from producing more samples. You can see this with
> gst-launch with:
> 
> gstbaseaudiosrc.c(838): gst_base_audio_src_create ():
> /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
> Dropped 37920 samples. This is most likely because downstream can't keep
> up and is consuming samples too slowly.
> 
> The reason is that the default max size of the queue is 1 second and you
> ask it to buffer 2 seconds at least. You need to increase the
> max-size-time to something bigger than 2 seconds. We should make queue
> error on these invalid configurations..

Sorry I just accidentally omitted the part (when writing this email) where I  
set max-size-time to 4 seconds and buffer and size limits to zero. So in fact 
the pipeline with which I tested did not have this error.
 
> It does not seem to cause and wrong skews here, though. Things just get
> choppy while the source drops samples and the sink consumes the
> occasional bursts of samples.

> > What happens occasionally is that audio gets choppy and then stops
> > completely. I have captured logs of one such incident, pulled out some
> > information from clock skewing and plotted it with gnuplot for which I
> > am attaching two plots.
> > 
> > On the full plot you can see that after the second large skew spike
> > average skew calculation seems to go wrong and starts oscillating
> > between positive and negative values. On the zoomed in view you can see
> > that there is no negative skew values, while average keeps oscillating.
> > However on the first occurence of such large spike it seems to keep the
> > average correct.
> > 
> > Basically I have two question here. First is whether wrong average skew
> > is expected to have negative effects on audio, including stopping
> > completely, and then secondly whether it looks like a bug in average
> > skew calculation which should be investigated deeper?
> 
> A wrong average skew could indeed mess up the timing.

Let's look at the part where it is going bad then:

<sample_num> <time> <skew> <avg_skew>
28504 0:05:12.396161786 20999999 18596044
28505 0:05:12.775732497 218666667 24848250

    sink->priv->avg_skew = (31 * sink->priv->avg_skew + skew) / 32;

(31 * 24848250 + 218666667 ) / 32 = 30905075

  mdrift = sink->priv->drift_tolerance * 1000; = 40000000;
  mdrift2 = mdrift / 2; = 20000000

  if (sink->priv->avg_skew > mdrift2) {
...
    sink->priv->avg_skew -= mdrift; = -9094925

avg_skew goes negative here, although strangely it does not match the next 
real data point. I assumed drift tolerance is a default 40ms, but maybe it can 
get auto-adjusted at runtime somehow? Or I just made an oversight in following 
the code and doing the arithmetic...

Anyway, avg_skew from now on oscillates:

28506 0:05:13.150555343 360000000 -3428257
28507 0:05:13.154222121 353500000 7725751
28508 0:05:13.154989086 353500000 18531196
28509 0:05:13.155440401 354333333 29025012
28510 0:05:13.194284943 310833333 -918477
28511 0:05:13.194986604 310833333 8823767

Even when this skew spike passes - samples 28600 onward.

Thanks,

Tvrtko