shmsink gets stuck

[Ross Currie]

Hi folks,

I'm experiencing an issue wherein a temporary load spike on a system causes
shmsink to stop delivering frames permanently.
(a cron job is doing it to me in real life, but I can replicate the issue
with "stress -m 5 -t 5")

My pipeline is decklinkvideosrc ! videoconvert ! shmsink    -> snowmix
(the issue can be duplicated using filesrc location=test.mp4 ! decodebin in
place of the decklinkvideosrc)

more precicely:
gst-launch-1.0 -v decklinkvideosrc mode=19 connection=sdi device-number=0
name=decoder ! videorate ! videoscale ! videoconvert ! video/x-raw,
format=\(string\)BGRA, pixel-aspect-ratio=\(fraction\)1/1,
interlace-mode=\(string\)progressive, width=\(int\)1280, height=\(int\)720,
framerate=\(fraction\)30/1 ! queue leaky=2 ! shmsink
socket-path=/tmp/feed1-control-pipe
shm-size=100000000 wait-for-connection=0 sync=false


So, my shm size is roughly 27 frames.

It appears that during the load spike frames pile up in the shm space.
When load returns to normal, the shm client reads and ACKs every one of the
backlogged buffers, but they do not seem to be properly freed in the
shmsink.
If everything recovers before the shm space is filled (say, 25 frames have
accumulated in the ShmAllocSpace) then everything seems to recover fine,
frames flow normally.

But if shmsink ever fills the ShmAllocSpace then I'm boned.  the client
will, when it recovers, read all the buffers that are in there, but no more
will ever be written.


When it ends up in this state, this function:  shm_alloc_space_alloc_block
  in shmalloc.c  is being called for each new frame and is returning NULL.
When this is happening, the ShmAllocSpace linked list of blocks has 27
entries.   It stays at 27 forever.

I watched  sp_writer_recv  in shmpipe.c  and I can see an ACK come in for
every one of those 27 buffers.  The ShmPipe's linked list of buffers get
decremented from 27 down to a normal level, but the ShmAllocSpace's list of
blocks never goes down.

I tried to understand how blocks in the ShmAllocSpace get released or
re-used but I'm failing to wrap my head around it.

Any thoughts would be welcome.
For the moment I can just throw a huge shm-size at the problem (300
frames?) and hope I never have a load issue that lasts that long.
That works but isn't a very satisfying solution.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20161231/1efdf449/attachment.html>