problem encoding h.264 and mpeg-ts

[Clarke, Trevor]

I've got an app that generates frames into multiple appsrc's, does some processing, encodes the frames to h.264 then muxes them all into an mpegts stream for transmission over udp multicast. Here's a summary of what it's doing (removed some custom plugins but they don't change caps):

appsrc is-live=true -> videoconvert -> x264enc  bframes=0 tune=zerolatency bitrate=2500 byte-stream=true -> queue -> mux.
appsrc is-live=true -> videoconvert -> x264enc  bframes=0 tune=zerolatency bitrate=2500 byte-stream=true -> queue -> mux.
... 10 appsrc
mux. is
mpegtsmux ! tee name=split ! queue2 ! chopmydata max-size=32700 ! udpsink sync=true host=224.16.10.20 port=10000 
split. ! queue2 ! multifilesink sync=true location=... next-file=key-unit-event port-messages=true

The x264enc is sometimes a custom nvenc wrapper. Mpegtsmux has been modified to include support for other metadata streams. But other than that, the above is a pretty close representation. The mpegtsmux is configured to set a different program for each stream. Some of the streams are VGA-ish sized, some are HD-ish sized, some are odd off-sized.

The problem I have is when I push test patterns with black and white alternating bars, black and white alternating checkerboards, and similar patterns which compress a lot I'm seeing good playback performance on different machines (mac and windows, server on linux, mostly using vlc for playback)

When I switch the test patterns to noisier data (randomized snow patterns) and the compression ratio drops I'm seeing significant playback issues to the point where is can take 30+ seconds just to see the program map show up in VLC and I get a couple of decoded frames per minute. Iftop on the server and client show that the raw packet throughput is roughly the same so I don't think I'm dropping packets. I ran ffprobe and ffplay on a mac client to see if it gave me any more information. Ffprobe shows a ton of errors:
"decode_slice_header error"
"non-existing PPS 0 referenced"
"no frame!"

This repeats for 15 seconds or so then I finally see the stream listing. Similar behavior in ffplay, lots of errors before syncing up to the stream then a frame here and there trickles through.

Since I'm not dropping packets I'm thinking it might be a problem on the encoding side. Any thoughts?
----------------------
Trevor R.H. Clarke
Software Engineer, Ball Aerospace
(937)320-7087




This message and any enclosures are intended only for the addressee.  Please 
notify the sender by email if you are not the intended recipient.  If you are 
not the intended recipient, you may not use, copy, disclose, or distribute this 
message or its contents or enclosures to any other person and any such actions 
may be unlawful.  Ball reserves the right to monitor and review all messages 
and enclosures sent to or from this email address.