How do you record audio and video off a webcam?

Sun Jun 17 10:44:02 UTC 2018

On Sat, Jun 16, 2018 at 08:17:18PM -0700, W.A. Garrett Weaver wrote:
> I want to take audio and video from a webcam and record it to an mp4 file. When
> I attempt to do this audio cuts out 5 seconds after the recording starts and
> gstreamer reports:
> >WARNING: from element /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0: Can't
> record audio fast enough
> >Dropped 22932 samples. This is most likely because downstream can't keep up
> and is consuming samples too slowly.

I had that too.

> I have the Logitech C920, which has the really nice feature of providing h.264
> video directly so the CPU doesn't need to compress it before writing it out to
> a file. I suspect that I am either missing something that needs to be in the
> pipeline to either either buffer out the streams, force everything to stay in
> sink or there is an option that needs to be added to mpegtsmux, but I don't
> know enough about gstreamer to know what would be. 
> 
> Here is the basic version of the pipeline I'm trying to employ:
> 
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width=
> 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 !
> audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux ! filesink
> location='output.mp4'

In my tests, I fixed it with two changes;

1.  add a videorate element after v4l2src, because a camera may not always generate a constant frame rate even if the source has negotiated to do so; although in my case the video was not already encoded, it was raw,

2.  increase the size of the queues before the mux, as the defaults were quite small,

3.  when autovideosink was also used, place a videorate ahead of it to limit the display framerate; you're not using a display sink, so this shouldn't matter for you.

I was also going to check that the audio data was being sampled at the correct rate; but didn't need to go that far.  I was working with prototype hardware where I wasn't at the time sure that 8000 samples at 8000 Hz would be one second long; i.e. bad audio codec clocks.

> Below are some observations from my experiments
> - The CPU is not the limiting factor. In fact, when I run the pipeline and run
> `top' and `gnome-system-monitor' the reported cpu usage does not go above 30%
> - The audio drops are not caused by the audio recording device. Using a
> `audiotestsrc' keeps frame drop warnings from being reported, but the output
> file still has audio dropping out.
> - The audio drops are not caused by the recording medium. Using a `fakesink' 
> - The audio drops are not dependent on high video resolutions, they happen with
> both 1080p video and 480p. 
> - Adding queues many places does not improve performance 
> - Changing the container and the codecs (moving to theora and matroska) removes
> these audio drop issues, but this has other consequences such as needing more
> of the CPU resources to compress the video. 
> 
> Here is my gstreamer version:
> >gst-launch-1.0 --version
> gst-launch-1.0 version 1.10.4
> GStreamer 1.10.4
> [1]http://packages.qa.debian.org/gstreamer1.0
> 
> Below are my experimental pipelines and the results observed.
> 
> # Record C920 video and audio to a file
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width=
> 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 !
> audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
> avenc_aac ! queue ! mux. mpegtsmux name=mux ! filesink location='output.mp4'
> # Audio cuts out after 5 seconds and we also got "Can't record audio fast
> enough", no drop in CPU usage 
> 
> # Put the video audio recording infront of the video for recording C920 video
> and audio to a file
> gst-launch-1.0 -v pulsesrc device=2 ! queue ! audio/x-raw,format=S16LE,channels
> =2,rate=8000 ! queue ! audioconvert ! queue ! avenc_aac ! queue ! mux. v4l2src
> device=/dev/video1 ! queue ! video/x-h264,width=1920,height=1080,framerate=30/1
> ! queue ! h264parse ! queue ! mux. mpegtsmux name=mux ! filesink location=
> 'output.mp4'
> # Audio cuts out after 5 seconds and we also got "Can't record audio fast
> enough", no drop in CPU usage 
> 
> # Try a different mux for recording C920 video and audio to a file
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width=
> 1920,height=1080,framerate=30/1 ! h264parse ! avdec_h264 ! theoraenc ! queue !
> mux. pulsesrc device=2 ! audio/x-raw,format=S16LE,channels=2,rate=8000 !
> audioconvert ! queue ! avenc_aac ! queue ! mux. matroskamux name=mux ! filesink
> location='output.mkv'
> # Audio and video was in sync for the whole video and we also got "Can't record
> audio fast enough", but video looked bad and CPU usage was huge
> 
> # Try the mp3 audio codec
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width=
> 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 !
> audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
> lamemp3enc ! queue ! mux. mpegtsmux name=mux ! filesink location='output.mp4'
> # Just flatout didn't start, perhaps I can only use AAC with the mp4 container
> 
> # Try a test src
> gst-launch-1.0 -v audiotestsrc ! queue ! audio/x-raw,format=S16LE,channels=
> 2,rate=8000 ! queue ! audioconvert ! queue ! avenc_aac ! queue ! mux. v4l2src
> device=/dev/video1 ! queue ! video/x-h264,width=1920,height=1080,framerate=30/1
> ! queue ! h264parse ! queue ! mux. mpegtsmux name=mux ! filesink location=
> 'output.mp4' 
> # Didn't get any frame drop errors but audio still drops out after 5 seconds
> 
> # Try using a fake sink
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width=
> 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 !
> audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
> avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! fakesink
> # We still got "Can't record audio fast enough"
> 
> # Try putting a queue between mpegtsmux and filesink
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width=
> 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2 !
> audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
> avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink location=
> 'output.mp4'
> # Audio drops out between 4 and 10 seconds and we also got "Can't record audio
> fast enough"
> 
> # add sync=false at the sink
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-h264,width=1920,height=
> 1080,framerate=30/1 ! queue ! h264parse ! queue ! mux. pulsesrc device=2 !
> audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert ! queue !
> avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink location=
> 'output.mp4' sync=false
> # Audio cuts out at 4 seconds and we also got "Can't record audio fast enough"
> 
> # Add provide-clock=false to pulsesrc from [2]https://stackoverflow.com/
> questions/35744114/getting-warning-messages-from-alsasrc
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue ! video/x-h264,width=
> 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc device=2
> provide-clock=false ! audio/x-raw,format=S16LE,channels=2,rate=8000 !
> audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue !
> filesink location='output.mp4' sync=false
> # Doesn't help, no difference in performance Audio cuts out at 4 seconds and we
> also got "Can't record audio fast enough"
> 
> # Changed some queues around
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-h264,width=1920,height=
> 1080,framerate=30/1 ! queue ! h264parse ! queue ! mux. pulsesrc device=2 !
> audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert ! queue !
> avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink location=
> 'output.mp4' sync=false
> # Audio falls out after 5 seconds we also got "Can't record audio fast enough",
> no difference in performance 
> 
> # try using "voaacenc bitrate=65536 ! aacparse" from [3]https://
> stackoverflow.com/questions/23505166/
> use-gstreamer-to-stream-video-and-audio-of-logitech-c920
> gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-h264,width=1920,height=
> 1080,framerate=30/1 ! queue ! h264parse ! queue ! mux. pulsesrc device=2 !
> audio/x-raw ! queue ! audioconvert ! queue ! voaacenc bitrate=65536 ! aacparse
> ! queue ! mux. mpegtsmux name=mux ! queue ! filesink location='output.mp4' sync
> =false
> # Audio falls out after 5 seconds we also got "Can't record audio fast enough",
> no difference in performance 
> 
> If you are intrigued by this issue please let me know your thoughts. 
> 
> References:
> 
> [1] http://packages.qa.debian.org/gstreamer1.0
> [2] https://stackoverflow.com/questions/35744114/getting-warning-messages-from-alsasrc
> [3] https://stackoverflow.com/questions/23505166/use-gstreamer-to-stream-video-and-audio-of-logitech-c920

> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

-- 
James Cameron
http://quozl.netrek.org/