Troubleshooting gst-launch freezes/hangs - how?

Fri Aug 14 10:31:34 PDT 2015

On 05 Aug 2015, at 12:56 AM, Thiago Santos <thiagoss at osg.samsung.com> wrote:

> You can use GST_DEBUG to generate logs of the run, but a run of hours can generate a massively large dump. You can try to guess which elements are involved to not have to log everything. Something like:
> 
> GST_DEBUG=2,*hls*:9,*mpeg*:9,GST_SCHEDULING:9 will give some information about the buffers flowing from pads to pads and also what the hls and mpeg elements are doing.

I have tried putting in the progressreport to get a sense of what data is flowing when gstreamer stops, and it turns out that when the hang occurs all data flow seems to stop dead:

A snapshot of what the threads are doing shows all the threads as stuck.

What seems to be happening is that at the point that things break, gst-launch prints out an additional set of caps, which implies that some part of gstreamer is either being initialised or reinitialised, is this correct?

Is there a way to figure out what is triggering the printing of the caps?

(gdb) info threads
  Id   Target Id         Frame 
  19   Thread 0x75efa470 (LWP 2489) "VCHIQ completio" 0x76b91dcc in ioctl ()
   from /lib/arm-linux-gnueabihf/libc.so.6
  18   Thread 0x756fa470 (LWP 2490) "HDispmanx Notif" 0x76c0b770 in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  17   Thread 0x74efa470 (LWP 2491) "HTV Notify" 0x76c0b770 in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  16   Thread 0x746fa470 (LWP 2492) "HCEC Notify" 0x76c0b770 in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  15   Thread 0x73eec470 (LWP 2493) "gst-launch-1.0" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  14   Thread 0x736ec470 (LWP 2494) "ILCS_HOST" 0x76c0b770 in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  13   Thread 0x72eec470 (LWP 2495) "queue4:src" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  12   Thread 0x724ff470 (LWP 2496) "queue2:src" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  11   Thread 0x71cff470 (LWP 2497) "queue1:src" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  10   Thread 0x714ff470 (LWP 2498) "queue3:src" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  9    Thread 0x70cff470 (LWP 2499) "queue0:src" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
---Type <return> to continue, or q <return> to quit--- 
  8    Thread 0x704ff470 (LWP 2500) "udpsrc0:src" 0x76b8c830 in poll ()
   from /lib/arm-linux-gnueabihf/libc.so.6
  7    Thread 0x6fcff470 (LWP 2501) "gmain" 0x76b8c830 in poll ()
   from /lib/arm-linux-gnueabihf/libc.so.6
  6    Thread 0x6f4ff470 (LWP 2502) "rtpsession-rtcp" 0x76b8ca28 in ppoll ()
   from /lib/arm-linux-gnueabihf/libc.so.6
  5    Thread 0x6ecff470 (LWP 2503) "timer" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  4    Thread 0x6e4ff470 (LWP 2504) "rtpjitterbuffer" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  3    Thread 0x6dcff470 (LWP 2505) "omxmpeg2videode" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
  2    Thread 0x6d38a470 (LWP 2506) "omxh264enc-omxh" 0x76c09504 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
* 1    Thread 0x76f35210 (LWP 2487) "gst-launch-1.0" 0x76b8c830 in poll ()
   from /lib/arm-linux-gnueabihf/libc.so.6

This is what is printed when the hang occurs:

progressreport0 (00:32:57): 1976 seconds
progressreport1 (00:32:57): 1976 seconds
progressreport2 (00:32:57): -1 seconds
progressreport0 (00:33:02): 1981 seconds
progressreport2 (00:33:02): 0 seconds
progressreport1 (00:33:02): 1981 seconds
/GstPipeline:pipeline0/GstMpegvParse:mpegvparse0.GstPad:src: caps = "video/mpeg\,\ mpegversion\=\(int\)2\,\ systemstream\=\(boolean\)false\,\ parsed\=\(boolean\)true\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ framerate\=\(fraction\)25/1\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ codec_data\=\(buffer\)000001b322024023249f238210202020262c342c2c262c2c3434363a363634362c34363a3a363a4050404436443a363034443a4a444c506050464c3a443a4a445c60745c46464c708a8a70a71012111113141714141316141819191c1a191718151617181a1b1d1d221e1d1a1b191715161a1b1c1d222124211f1f1c1a1b1f20242326231e2021262928272b000001b5148200010000\,\ profile\=\(string\)main\,\ level\=\(string\)main\,\ interlace-mode\=\(string\)mixed"
/GstPipeline:pipeline0/GstOMXMPEG2VideoDec-omxmpeg2videodec:omxmpeg2videodec-omxmpeg2videodec0.GstPad:sink: caps = "video/mpeg\,\ mpegversion\=\(int\)2\,\ systemstream\=\(boolean\)false\,\ parsed\=\(boolean\)true\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ framerate\=\(fraction\)25/1\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ codec_data\=\(buffer\)000001b322024023249f238210202020262c342c2c262c2c3434363a363634362c34363a3a363a4050404436443a363034443a4a444c506050464c3a443a4a445c60745c46464c708a8a70a71012111113141714141316141819191c1a191718151617181a1b1d1d221e1d1a1b191715161a1b1c1d222124211f1f1c1a1b1f20242326231e2021262928272b000001b5148200010000\,\ profile\=\(string\)main\,\ level\=\(string\)main\,\ interlace-mode\=\(string\)mixed"
/GstPipeline:pipeline0/GstOMXMPEG2VideoDec-omxmpeg2videodec:omxmpeg2videodec-omxmpeg2videodec0.GstPad:src: caps = "video/x-raw\(memory:EGLImage\)\,\ format\=\(string\)RGBA\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstOMXMPEG2VideoDec-omxmpeg2videodec:omxmpeg2videodec-omxmpeg2videodec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)1:1:0:0\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstOMXMPEG2VideoDec-omxmpeg2videodec:omxmpeg2videodec-omxmpeg2videodec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstProgressReport:progressreport1.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstProgressReport:progressreport1.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstVideoScale:videoscale0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)384\,\ height\=\(int\)256\,\ pixel-aspect-ratio\=\(fraction\)8/9\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)384\,\ height\=\(int\)256\,\ pixel-aspect-ratio\=\(fraction\)8/9\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstVideoRate:videorate0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)384\,\ height\=\(int\)256\,\ pixel-aspect-ratio\=\(fraction\)8/9\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstOMXH264Enc-omxh264enc:omxh264enc-omxh264enc0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)384\,\ height\=\(int\)256\,\ pixel-aspect-ratio\=\(fraction\)8/9\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstVideoRate:videorate0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)384\,\ height\=\(int\)256\,\ pixel-aspect-ratio\=\(fraction\)8/9\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)384\,\ height\=\(int\)256\,\ pixel-aspect-ratio\=\(fraction\)8/9\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
/GstPipeline:pipeline0/GstVideoScale:videoscale0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)I420\,\ width\=\(int\)544\,\ height\=\(int\)576\,\ pixel-aspect-ratio\=\(fraction\)24/17\,\ interlace-mode\=\(string\)mixed\,\ colorimetry\=\(string\)bt601\,\ framerate\=\(fraction\)25/1"
progressreport0 (00:33:07): 1986 seconds

The pipeline looks like this:

gst-launch-1.0 -v udpsrc uri=udp://[ff08::30]:5004 "caps=application/x-rtp,media=(string)video,clock-rate=(int)90000" ! rtpbin ! rtpmp2tdepay ! progressreport update-freq=5 ! tsparse ! tsdemux name=demux ! queue ! mpegvideoparse ! omxmpeg2videodec ! progressreport update-freq=5 ! queue ! videoscale ! video/x-raw,width=384,height=256 ! videorate ! omxh264enc target-bitrate=200000 control-rate=variable inline-header=true ! queue ! progressreport update-freq=5 ! mpegtsmux name=mux ! hlssink location=/var/www/stream/segment2%05d.ts playlist-location=/var/www/stream/output2.m3u8 playlist-root=http://192.168.225.3/stream/ target-duration=1 demux. ! queue ! mpegaudioparse ! mpg123audiodec ! audiorate ! voaacenc ! aacparse ! queue ! mux.

Regards,
Graham
—