<div dir="ltr">James, Nicolas, <div> </div><div>Thank you for your replies. I have increased the queue length after the h264parse and that enables the pipeline to record video and audio well beyond the 5 second limit being experienced before. Below is the final pipeline. Notice I have removed the formatting of the audio and the device id changed when I removed the webcam and plugged it back it. </div><div> </div><div>gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! queue max-size-buffers=0 max-size-bytes=0 max-size-time=1000000000 ! mux. pulsesrc device=7 ! audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux ! filesink location='output.mp4' </div><div> </div><div>This problem is now resolved. </div></div><div class="gmail_extra"> <div class="gmail_quote">On Sun, Jun 17, 2018 at 10:54 AM, Nicolas Dufresne <<a href="mailto:nicolas@ndufresne.ca" target="_blank">nicolas@ndufresne.ca</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Le dimanche 17 juin 2018 à 20:44 +1000, James Cameron a écrit : > On Sat, Jun 16, 2018 at 08:17:18PM -0700, W.A. Garrett Weaver wrote: > > I want to take audio and video from a webcam and record it to an mp4 file. When > > I attempt to do this audio cuts out 5 seconds after the recording starts and > > gstreamer reports: > > > WARNING: from element /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0: Can't > > > > record audio fast enough > > > Dropped 22932 samples. This is most likely because downstream can't keep up > > > > and is consuming samples too slowly. > > I had that too. > > > I have the Logitech C920, which has the really nice feature of providing h.264 > > video directly so the CPU doesn't need to compress it before writing it out to > > a file. I suspect that I am either missing something that needs to be in the > > pipeline to either either buffer out the streams, force everything to stay in > > sink or there is an option that needs to be added to mpegtsmux, but I don't > > know enough about gstreamer to know what would be. > > > > Here is the basic version of the pipeline I'm trying to employ: > > > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width= > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 ! > > audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux ! filesink > > location='output.mp4' > > In my tests, I fixed it with two changes; > > 1. add a videorate element after v4l2src, because a camera may not always generate a constant frame rate even if the source has negotiated to do so; although in my case the video was not already encoded, it was raw, This is not applicable in this context, because videorate won't accept H264. > > 2. increase the size of the queues before the mux, as the defaults were quite small, This is likely what is happening. The queue size won't old the HW initialization delay of the C920 (which can be up to 5s). Dropping any H264 frames will directly lead to broken video. Have a try the following configuration: v4l2src ! h264parse ! queue max-size-buffers=0 max-size-bytes=0 max- size-time=1000000000 ! ... If you have GStreamer <= 1.12, you might need even longer queue duration. <div class="HOEnZb"><div class="h5"> > > 3. when autovideosink was also used, place a videorate ahead of it to limit the display framerate; you're not using a display sink, so this shouldn't matter for you. > > I was also going to check that the audio data was being sampled at the correct rate; but didn't need to go that far. I was working with prototype hardware where I wasn't at the time sure that 8000 samples at 8000 Hz would be one second long; i.e. bad audio codec clocks. > > > Below are some observations from my experiments > > - The CPU is not the limiting factor. In fact, when I run the pipeline and run > > `top' and `gnome-system-monitor' the reported cpu usage does not go above 30% > > - The audio drops are not caused by the audio recording device. Using a > > `audiotestsrc' keeps frame drop warnings from being reported, but the output > > file still has audio dropping out. > > - The audio drops are not caused by the recording medium. Using a `fakesink' > > - The audio drops are not dependent on high video resolutions, they happen with > > both 1080p video and 480p. > > - Adding queues many places does not improve performance > > - Changing the container and the codecs (moving to theora and matroska) removes > > these audio drop issues, but this has other consequences such as needing more > > of the CPU resources to compress the video. > > > > Here is my gstreamer version: > > > gst-launch-1.0 --version > > > > gst-launch-1.0 version 1.10.4 > > GStreamer 1.10.4 > > [1]<a href="http://packages.qa.debian.org/gstreamer1.0" rel="noreferrer" target="_blank">http://packages.qa.debian.org/gstreamer1.0</a> > > > > Below are my experimental pipelines and the results observed. > > > > # Record C920 video and audio to a file > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width= > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 ! > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue ! > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! filesink location='output.mp4' > > # Audio cuts out after 5 seconds and we also got "Can't record audio fast > > enough", no drop in CPU usage > > > > # Put the video audio recording infront of the video for recording C920 video > > and audio to a file > > gst-launch-1.0 -v pulsesrc device=2 ! queue ! audio/x-raw,format=S16LE,channels > > =2,rate=8000 ! queue ! audioconvert ! queue ! avenc_aac ! queue ! mux. v4l2src > > device=/dev/video1 ! queue ! video/x-h264,width=1920,height=1080,framerate=30/1 > > ! queue ! h264parse ! queue ! mux. mpegtsmux name=mux ! filesink location= > > 'output.mp4' > > # Audio cuts out after 5 seconds and we also got "Can't record audio fast > > enough", no drop in CPU usage > > > > # Try a different mux for recording C920 video and audio to a file > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width= > > 1920,height=1080,framerate=30/1 ! h264parse ! avdec_h264 ! theoraenc ! queue ! > > mux. pulsesrc device=2 ! audio/x-raw,format=S16LE,channels=2,rate=8000 ! > > audioconvert ! queue ! avenc_aac ! queue ! mux. matroskamux name=mux ! filesink > > location='output.mkv' > > # Audio and video was in sync for the whole video and we also got "Can't record > > audio fast enough", but video looked bad and CPU usage was huge > > > > # Try the mp3 audio codec > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width= > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 ! > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue ! > > lamemp3enc ! queue ! mux. mpegtsmux name=mux ! filesink location='output.mp4' > > # Just flatout didn't start, perhaps I can only use AAC with the mp4 container > > > > # Try a test src > > gst-launch-1.0 -v audiotestsrc ! queue ! audio/x-raw,format=S16LE,channels= > > 2,rate=8000 ! queue ! audioconvert ! queue ! avenc_aac ! queue ! mux. v4l2src > > device=/dev/video1 ! queue ! video/x-h264,width=1920,height=1080,framerate=30/1 > > ! queue ! h264parse ! queue ! mux. mpegtsmux name=mux ! filesink location= > > 'output.mp4' > > # Didn't get any frame drop errors but audio still drops out after 5 seconds > > > > # Try using a fake sink > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width= > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 ! > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue ! > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! fakesink > > # We still got "Can't record audio fast enough" > > > > # Try putting a queue between mpegtsmux and filesink > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width= > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 ! > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue ! > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink location= > > 'output.mp4' > > # Audio drops out between 4 and 10 seconds and we also got "Can't record audio > > fast enough" > > > > # add sync=false at the sink > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-h264,width=1920,height= > > 1080,framerate=30/1 ! queue ! h264parse ! queue ! mux. pulsesrc device=2 ! > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert ! queue ! > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink location= > > 'output.mp4' sync=false > > # Audio cuts out at 4 seconds and we also got "Can't record audio fast enough" > > > > # Add provide-clock=false to pulsesrc from [2]<a href="https://stackoverflow.com/" rel="noreferrer" target="_blank">https://stackoverflow.com/</a> > > questions/35744114/getting-warning-messages-from-alsasrc > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width= > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 > > provide-clock=false ! audio/x-raw,format=S16LE,channels=2,rate=8000 ! > > audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! > > filesink location='output.mp4' sync=false > > # Doesn't help, no difference in performance Audio cuts out at 4 seconds and we > > also got "Can't record audio fast enough" > > > > # Changed some queues around > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-h264,width=1920,height= > > 1080,framerate=30/1 ! queue ! h264parse ! queue ! mux. pulsesrc device=2 ! > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert ! queue ! > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink location= > > 'output.mp4' sync=false > > # Audio falls out after 5 seconds we also got "Can't record audio fast enough", > > no difference in performance > > > > # try using "voaacenc bitrate=65536 ! aacparse" from [3]https:// > > <a href="http://stackoverflow.com/questions/23505166/" rel="noreferrer" target="_blank">stackoverflow.com/questions/23505166/</a> > > use-gstreamer-to-stream-video-and-audio-of-logitech-c920 > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-h264,width=1920,height= > > 1080,framerate=30/1 ! queue ! h264parse ! queue ! mux. pulsesrc device=2 ! > > audio/x-raw ! queue ! audioconvert ! queue ! voaacenc bitrate=65536 ! aacparse > > ! queue ! mux. mpegtsmux name=mux ! queue ! filesink location='output.mp4' sync > > =false > > # Audio falls out after 5 seconds we also got "Can't record audio fast enough", > > no difference in performance > > > > If you are intrigued by this issue please let me know your thoughts. > > > > References: > > > > [1] <a href="http://packages.qa.debian.org/gstreamer1.0" rel="noreferrer" target="_blank">http://packages.qa.debian.org/gstreamer1.0</a> > > [2] <a href="https://stackoverflow.com/questions/35744114/getting-warning-messages-from-alsasrc" rel="noreferrer" target="_blank">https://stackoverflow.com/questions/35744114/getting-warning-messages-from-alsasrc</a> > > [3] <a href="https://stackoverflow.com/questions/23505166/use-gstreamer-to-stream-video-and-audio-of-logitech-c920" rel="noreferrer" target="_blank">https://stackoverflow.com/questions/23505166/use-gstreamer-to-stream-video-and-audio-of-logitech-c920</a> > > _______________________________________________ > > gstreamer-devel mailing list > > <a href="mailto:gstreamer-devel@lists.freedesktop.org">gstreamer-devel@lists.freedesktop.org</a> > > <a href="https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a> > > _______________________________________________ gstreamer-devel mailing list <a href="mailto:gstreamer-devel@lists.freedesktop.org">gstreamer-devel@lists.freedesktop.org</a> <a href="https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel" rel="noreferrer" target="_blank">https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel</a> </div></div></blockquote></div> <div> </div>-- <div class="gmail_signature" data-smartmail="gmail_signature"><div>W.A. Garrett Weaver </div> <div><a href="mailto:weaverg@email.arizona.edu" target="_blank">weaverg@email.arizona.edu</a></div></div> </div>