[pulseaudio-discuss] Crackling audio with Pulseaudio 4.0 and the simple Pulse API.

[Tanu Kaskinen]

On Thu, 2013-07-11 at 21:54 +0600, Alexander E. Patrakov wrote:
> 2013/7/11 Tanu Kaskinen <tanu.kaskinen at linux.intel.com>:
> > You don't understand me, and I don't understand you... What is the
> > result of the boundary being off? Does audio get skipped, duplicated, or
> > is there no error at all?
> 
> No duration-related error at all in my understanding, see below where
> it differs from yours.
> 
> > I'll try to clarify this (also to myself) with an example. Let's have a
> > sink, and a stream with a resampler in between. For simplicity, let's
> > assume that the resampler doesn't actually do any resampling, so when
> > the sink asks for 10 samples, the resampler reads from the stream 10
> > samples.
> >
> > Let's say that the write index of the sink is N, and the resampler has
> > one sample buffered. Due to the buffered sample, the read index of the
> > stream is N+1.
> 
> Let me rephrase this in order to check that I understand. You are
> talking about a resampler that has 1:1 input:output sample rate ratio.
> The resampler needs to look by one sample ahead (or behind, depending
> on how you look at it) in order to function. A simple example of such
> "resampler" would be something that averages each incoming sample with
> the previous one.
> 
> You have pushed 11 samples into the resampler. You say that the
> resampler has consumed one sample for its internal buffer, consumed 10
> more samples "for good output" and produced 10 output samples. And
> this is the point I don't quite agree with.
> 
> What you describe is one possible behaviour (and I'd say a buggy one,
> but it does exist), but we need to consider one more possibility. The
> other case is that the resampler produces 11 samples when fed 11
> samples. The first sample in my example is the average of zero and the
> first input sample, and it's technically wrong to throw it out,
> because this would mean a change in the zero-extended output from
> prepending an all-zero sequence to the input. I.e. resamplers should
> by default, at the beginning of the stream, treat internal buffers not
> as empty, but as full, pre-filled with zeros. See also the note at the
> end of this mail.
> 
> But let's say that, depending on the implementation, the read index
> might be either N or N+1.
> 
> > Now the sink is rewound by 10 samples. This means that the sink will
> > want the next written sample to be from index N-10. The resampler drops
> > the buffered sample, and the read index of the stream moves back by 10
> > samples to N-9. The sample at N-10 got lost, the user hears audio
> > skipping by one sample.
> 
> "the read index of the stream moves back by 10 samples to N-9" is of
> course wrong, as you point out below.
> 
> > The amount of dropped audio in the resampler buffer should have been
> > added to the amount by which the stream read index was moved back.
> 
> Sure. The confusion actually comes from your attempt to second-guess
> behind the resampler's back which input samples it wants due to a
> rewind. Actually, for non-1:1 resamplers, the amount of buffering is
> variable in time. E.g. consider a 3:2 downsampler that works by linear
> interpolation:
> 
> Y[0] = X[0]
> Y[1] = (X[1] + X[2]) / 2
> Y[2] = X[3]
> Y[3] = (X[4] + X[5]) / 2
> Y[4] = X[6]
> Y[5] = (X[7] + X[8]) / 2
> and so on
> 
> Sometimes, when fed a sample, it can copy the sample to the output,
> and sometimes it will average two neighbouring samples. Sure you can
> second-guess after this particular resampling pattern, but now
> consider another valid case of a linear-interpolation 3:2 downsampler:
> 
> Y[0] = (3 * X[0] + X[1]) / 4
> Y[1] = (X[1] + 3 * X[2]) / 4
> Y[2] = (3 * X[3] + X[4]) / 4
> Y[3] = (X[4] + 3 * X[5]) / 4
> Y[4] = (3 * X[6] + X[7]) / 4
> Y[5] = (X[7] + 3 * X[8]) / 4
> and so on
> 
> which always has to look ahead. Same maximum "buffer" length, same
> resample ratio, different input needs. E.g., in the first case, you
> have to know the input up to X[3] to determine Y[2], while in the
> second case you need to know one more input sample.
> 
> So let me repeat - don't attempt to guess which input samples the
> resampler will need after a sink rewind, you always will be wrong. Let
> the resampler implementation decide (i.e. you just have to implement a
> "pull" model instead of "push" if you allow arbitrary sink-based
> rewinds that need a rerun of the resampler), this naturally leads to
> the need to forward all rewind requests to the particular
> implementation. But see below for an alternative that you have
> correctly suggested, based on snapshots.

I think we are talking about different things. I was talking
specifically about the handling of the leftover buffer that we currently
have in pa_resampler. If the resampler backend does buffering, which you
seem to be talking about, that's a different (although in some sense
very similar) problem to solve, and I think we have an agreement about
the way to deal with it with state snapshots. (Nobody has volunteered to
implement the solution, though, but it's still good to have a plan.)

> > "The initial phase" means the last output sample relative to the new
> > position, right?
> 
> Yes. In the above 3:2 examples, it means whether an even or an odd
> output sample is produced. And if you look carefully, you will notice
> that the two examples above differ only by the shift by 1/4 input
> sample, so that can also be counted as a phase difference.
> 
> > It might be feasible to add the required functionality to the stock
> > resamplers. When we discussed the filter rewinding, you mentioned the
> > idea of maintaining a history of filter state snapshots. Taking a
> > snapshot only requires a function for copying the filter state, and I
> > would guess that adding such function to the stock resamplers could very
> > well be done.
> 
> True. It would also be convenient to store the read index of the
> stream and the write index of the sink along with the snapshots. This
> way, you just search for the latest snapshot that happened before the
> piece of the history that you want to amend, and continue from there,
> and this works for arbitrary buffering requirements of the resampler.

Ack.

-- 
Tanu