Drop frames if a filter (neural networks) is too slow

[MyungJoo Ham]

Dear Gstreamer Developers,


I'm developing gstreamer filters that either use general neural
network models as media filters or help such filters (transforming,
mux/demuxing, or converting tensors).

https://github.com/nnsuite/nnstreamer


One concern is that we have a lot of usage cases with
heavy neural networks (e.g., latencies of over 100ms and fluctuating)
on live video streams from cameras and we want to drop old-pending
video frames if there is a new video frame is coming while the filter
is still processing previous frame. (but not dropping already-being-processed
frames)


In other words, in a stream like this:

Camera(v4l2) --> Neural Network (tensor_converter + tensor_filter) --> sink

, let's assume that Camera is operating at 60FPS and
Neural Network is processing at 1FPS (although it's not realistic enough
to say "xxFPS" on these networks as they fluctuate a lot)

Then, we want to process 0th camera frame and 60th camera frame, and then
120th camera frame, .. and so on.

With common configurations, with large queues, it processes 0th, 1st, 2nd frame
and drops newer frames, not older frames if the queue is full.

Could you please enlightenme on which document to look at or
which part to implement for this matter?


Cheers,
MyungJoo