How do you record audio and video off a webcam?

[W.A. Garrett Weaver]

I want to take audio and video from a webcam and record it to an mp4 file.
When I attempt to do this audio cuts out 5 seconds after the recording
starts and gstreamer reports:
>WARNING: from element /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0: Can't
record audio fast enough
>Dropped 22932 samples. This is most likely because downstream can't keep
up and is consuming samples too slowly.

I have the Logitech C920, which has the really nice feature of providing
h.264 video directly so the CPU doesn't need to compress it before writing
it out to a file. I suspect that I am either missing something that needs
to be in the pipeline to either either buffer out the streams, force
everything to stay in sink or there is an option that needs to be added to
mpegtsmux, but I don't know enough about gstreamer to know what would be.

Here is the basic version of the pipeline I'm trying to employ:

gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! queue !
mux. pulsesrc device=2 ! audioconvert ! queue ! avenc_aac ! queue ! mux.
mpegtsmux name=mux ! filesink location='output.mp4'


Below are some observations from my experiments
- The CPU is not the limiting factor. In fact, when I run the pipeline and
run `top' and `gnome-system-monitor' the reported cpu usage does not go
above 30%
- The audio drops are not caused by the audio recording device. Using a
`audiotestsrc' keeps frame drop warnings from being reported, but the
output file still has audio dropping out.
- The audio drops are not caused by the recording medium. Using a
`fakesink'
- The audio drops are not dependent on high video resolutions, they happen
with both 1080p video and 480p.
- Adding queues many places does not improve performance
- Changing the container and the codecs (moving to theora and matroska)
removes these audio drop issues, but this has other consequences such as
needing more of the CPU resources to compress the video.

Here is my gstreamer version:
>gst-launch-1.0 --version
gst-launch-1.0 version 1.10.4
GStreamer 1.10.4
http://packages.qa.debian.org/gstreamer1.0


Below are my experimental pipelines and the results observed.

# Record C920 video and audio to a file
gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! queue !
mux. pulsesrc device=2 ! audio/x-raw,format=S16LE,channels=2,rate=8000 !
audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux !
filesink location='output.mp4'
# Audio cuts out after 5 seconds and we also got "Can't record audio fast
enough", no drop in CPU usage

# Put the video audio recording infront of the video for recording C920
video and audio to a file
gst-launch-1.0 -v pulsesrc device=2 ! queue !
audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert !
queue ! avenc_aac ! queue ! mux. v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! queue ! h264parse !
queue ! mux. mpegtsmux name=mux ! filesink location='output.mp4'
# Audio cuts out after 5 seconds and we also got "Can't record audio fast
enough", no drop in CPU usage

# Try a different mux for recording C920 video and audio to a file
gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! avdec_h264
! theoraenc ! queue ! mux. pulsesrc device=2 !
audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
avenc_aac ! queue ! mux. matroskamux name=mux ! filesink
location='output.mkv'
# Audio and video was in sync for the whole video and we also got "Can't
record audio fast enough", but video looked bad and CPU usage was huge

# Try the mp3 audio codec
gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! queue !
mux. pulsesrc device=2 ! audio/x-raw,format=S16LE,channels=2,rate=8000 !
audioconvert ! queue ! lamemp3enc ! queue ! mux. mpegtsmux name=mux !
filesink location='output.mp4'
# Just flatout didn't start, perhaps I can only use AAC with the mp4
container

# Try a test src
gst-launch-1.0 -v audiotestsrc ! queue !
audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert !
queue ! avenc_aac ! queue ! mux. v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! queue ! h264parse !
queue ! mux. mpegtsmux name=mux ! filesink location='output.mp4'
# Didn't get any frame drop errors but audio still drops out after 5 seconds

# Try using a fake sink
gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! queue !
mux. pulsesrc device=2 ! audio/x-raw,format=S16LE,channels=2,rate=8000 !
audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue
! fakesink
# We still got "Can't record audio fast enough"

# Try putting a queue between mpegtsmux and filesink
gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! queue !
mux. pulsesrc device=2 ! audio/x-raw,format=S16LE,channels=2,rate=8000 !
audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue
! filesink location='output.mp4'
# Audio drops out between 4 and 10 seconds and we also got "Can't record
audio fast enough"

# add sync=false at the sink
gst-launch-1.0 -v v4l2src device=/dev/video1 !
video/x-h264,width=1920,height=1080,framerate=30/1 ! queue ! h264parse !
queue ! mux. pulsesrc device=2 !
audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert !
queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink
location='output.mp4' sync=false
# Audio cuts out at 4 seconds and we also got "Can't record audio fast
enough"

# Add provide-clock=false to pulsesrc from
https://stackoverflow.com/questions/35744114/getting-warning-messages-from-alsasrc
gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! queue !
mux. pulsesrc device=2 provide-clock=false !
audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink
location='output.mp4' sync=false
# Doesn't help, no difference in performance Audio cuts out at 4 seconds
and we also got "Can't record audio fast enough"

# Changed some queues around
gst-launch-1.0 -v v4l2src device=/dev/video1 !
video/x-h264,width=1920,height=1080,framerate=30/1 ! queue ! h264parse !
queue ! mux. pulsesrc device=2 !
audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert !
queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink
location='output.mp4' sync=false
# Audio falls out after 5 seconds we also got "Can't record audio fast
enough", no difference in performance

# try using "voaacenc bitrate=65536 ! aacparse" from
https://stackoverflow.com/questions/23505166/use-gstreamer-to-stream-video-and-audio-of-logitech-c920
gst-launch-1.0 -v v4l2src device=/dev/video1 !
video/x-h264,width=1920,height=1080,framerate=30/1 ! queue ! h264parse !
queue ! mux. pulsesrc device=2 ! audio/x-raw ! queue ! audioconvert ! queue
! voaacenc bitrate=65536 ! aacparse ! queue ! mux. mpegtsmux name=mux !
queue ! filesink location='output.mp4' sync=false
# Audio falls out after 5 seconds we also got "Can't record audio fast
enough", no difference in performance

If you are intrigued by this issue please let me know your thoughts.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180616/4fdf383e/attachment-0001.html>