AW: Drop frames if a filter (neural networks) is too slow

[MyungJoo Ham]

Hi.

Oh.. it appears that with an additional queue element in front of a neural network filter, it's going to work! Thanks!

However, is there a way to do this without adding a queue between elements?
Maybe because we need a new thread to do this, it's going to be "NO" if we want simple solutions, I guess.

Thanks so much!

Cheers,
MyungJoo
 
--------- Original Message ---------
Sender : Thornton, Keith <keith.thornton at zeiss.com>
Date   : 2018-07-30 15:28 (GMT+9)
Title  : AW: Drop frames if a filter (neural networks) is too slow
 
Hi, have you tried a queue with max-size-buffers=1,max-size-bytes=0,max-size-time=0,leaky=2 
 
-----Ursprüngliche Nachricht-----
Von: gstreamer-devel [mailto:gstreamer-devel-bounces at lists.freedesktop.org] Im Auftrag von MyungJoo Ham
Gesendet: Montag, 30. Juli 2018 02:54
An: gstreamer-devel at lists.freedesktop.org
Cc: JIJOONG MOON <jijoong.moon at samsung.com>; Geunsik Lim <geunsik.lim at samsung.com>; Wook Song <wook16.song at samsung.com>; Jaeyun Jung <jy1210.jung at samsung.com>; Sangjung Woo <sangjung.woo at samsung.com>; Hyoungjoo Ahn <hello.ahn at samsung.com>; Jinhyuck Park <jinhyuck83.park at samsung.com>
Betreff: Drop frames if a filter (neural networks) is too slow
 
 
Dear Gstreamer Developers,
 
 
I'm developing gstreamer filters that either use general neural network models as media filters or help such filters (transforming, mux/demuxing, or converting tensors).
 
https://github.com/nnsuite/nnstreamer
 
 
One concern is that we have a lot of usage cases with heavy neural networks (e.g., latencies of over 100ms and fluctuating) on live video streams from cameras and we want to drop old-pending video frames if there is a new video frame is coming while the filter is still processing previous frame. (but not dropping already-being-processed
frames)
 
 
In other words, in a stream like this:
 
Camera(v4l2) --> Neural Network (tensor_converter + tensor_filter) --> sink
 
, let's assume that Camera is operating at 60FPS and Neural Network is processing at 1FPS (although it's not realistic enough to say "xxFPS" on these networks as they fluctuate a lot)
 
Then, we want to process 0th camera frame and 60th camera frame, and then 120th camera frame, .. and so on.
 
With common configurations, with large queues, it processes 0th, 1st, 2nd frame and drops newer frames, not older frames if the queue is full.
 
Could you please enlightenme on which document to look at or which part to implement for this matter?
 
 
Cheers,
MyungJoo
_______________________________________________
gstreamer-devel mailing list
gstreamer-devel at lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
 
 
--
MyungJoo Ham (함명주), Ph.D.
Autonomous Machine Lab., AI Center, Samsung Research.
Cell: +82-10-6714-2858