Time distortion on WebRTC video streams

[Guillaume Denis]

Hello,

I am encountering an issue with a server side processing of WebRTC video 
streams: an added/variable latency appears and increases over time (but 
I don't notice any obvious "slow motion" effect).

The pipeline is defined as:
appsrc format=time is-live=true do-timestamp=true name=src ! 
application/x-rtp, encoding-name=VP8-DRAFT-IETF-01 ! queue ! rtpvp8depay 
! decodebin ! videoconvert ! identity ! videoconvert ! vp8enc 
error-resilient=partitions keyframe-max-dist=10 auto-alt-ref=true 
cpu-used=5 deadline=1 ! rtpvp8pay ! appsink name=sink

(the intent is to replace identity by other effects)

The processing is done in parallel on two video streams:
peer A webcam from browser -> webrtc app -> GStreamer pipeline -> webrtc 
app -> peer B screen in browser
peer B webcam from browser -> webrtc app -> GStreamer pipeline -> webrtc 
app -> peer A screen in browser

The webrtc app is made with the pion/webrtc library (for 
signaling/connecting/routing of streams...), and an additional 
processing is done (with GStreamer) on audio streams, without any time 
distortion.

I have two ideas:

* either the entire processing (vp8 decoding/encoding to start with) is 
expensive enough to "break" real-time and introduce time distortions. I 
wonder if in this case there is a way to adapt the processing (for 
instance adjusting the framerate) and try to maintain a near constant 
latency
* there is a problem with how RTP packets are "time labeled" (with pts 
for instance) and synced

I would greatly appreciate any help, including how to debug/specify more 
precisely the encountered behavior and where it occurs.

Many thanks,
Guillaume