How do you record audio and video off a webcam?

Mon Jun 18 06:51:27 UTC 2018

James, Nicolas,

Thank you for your replies. I have increased the queue length after the
h264parse and that enables the pipeline to record video and audio well
beyond the 5 second limit being experienced before. Below is the final
pipeline. Notice I have removed the formatting of the audio and the device
id changed when I removed the webcam and plugged it back it.

gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! queue
max-size-buffers=0 max-size-bytes=0 max-size-time=1000000000 ! mux.
pulsesrc device=7 ! audioconvert ! queue ! avenc_aac ! queue ! mux.
mpegtsmux name=mux ! filesink location='output.mp4'

This problem is now resolved.

On Sun, Jun 17, 2018 at 10:54 AM, Nicolas Dufresne <nicolas at ndufresne.ca>
wrote:

> Le dimanche 17 juin 2018 à 20:44 +1000, James Cameron a écrit :
> > On Sat, Jun 16, 2018 at 08:17:18PM -0700, W.A. Garrett Weaver wrote:
> > > I want to take audio and video from a webcam and record it to an mp4
> file. When
> > > I attempt to do this audio cuts out 5 seconds after the recording
> starts and
> > > gstreamer reports:
> > > > WARNING: from element /GstPipeline:pipeline0/GstPulseSrc:pulsesrc0:
> Can't
> > >
> > > record audio fast enough
> > > > Dropped 22932 samples. This is most likely because downstream can't
> keep up
> > >
> > > and is consuming samples too slowly.
> >
> > I had that too.
> >
> > > I have the Logitech C920, which has the really nice feature of
> providing h.264
> > > video directly so the CPU doesn't need to compress it before writing
> it out to
> > > a file. I suspect that I am either missing something that needs to be
> in the
> > > pipeline to either either buffer out the streams, force everything to
> stay in
> > > sink or there is an option that needs to be added to mpegtsmux, but I
> don't
> > > know enough about gstreamer to know what would be.
> > >
> > > Here is the basic version of the pipeline I'm trying to employ:
> > >
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
> video/x-h264,width=
> > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc
> device=2 !
> > > audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux !
> filesink
> > > location='output.mp4'
> >
> > In my tests, I fixed it with two changes;
> >
> > 1.  add a videorate element after v4l2src, because a camera may not
> always generate a constant frame rate even if the source has negotiated to
> do so; although in my case the video was not already encoded, it was raw,
>
> This is not applicable in this context, because videorate won't accept
> H264.
>
> >
> > 2.  increase the size of the queues before the mux, as the defaults were
> quite small,
>
> This is likely what is happening. The queue size won't old the HW
> initialization delay of the C920 (which can be up to 5s). Dropping any
> H264 frames will directly lead to broken video. Have a try the
> following configuration:
>
>   v4l2src ! h264parse ! queue max-size-buffers=0 max-size-bytes=0 max-
> size-time=1000000000 ! ...
>
> If you have GStreamer <= 1.12, you  might need even longer queue
> duration.
>
> >
> > 3.  when autovideosink was also used, place a videorate ahead of it to
> limit the display framerate; you're not using a display sink, so this
> shouldn't matter for you.
> >
> > I was also going to check that the audio data was being sampled at the
> correct rate; but didn't need to go that far.  I was working with prototype
> hardware where I wasn't at the time sure that 8000 samples at 8000 Hz would
> be one second long; i.e. bad audio codec clocks.
> >
> > > Below are some observations from my experiments
> > > - The CPU is not the limiting factor. In fact, when I run the pipeline
> and run
> > > `top' and `gnome-system-monitor' the reported cpu usage does not go
> above 30%
> > > - The audio drops are not caused by the audio recording device. Using a
> > > `audiotestsrc' keeps frame drop warnings from being reported, but the
> output
> > > file still has audio dropping out.
> > > - The audio drops are not caused by the recording medium. Using a
> `fakesink'
> > > - The audio drops are not dependent on high video resolutions, they
> happen with
> > > both 1080p video and 480p.
> > > - Adding queues many places does not improve performance
> > > - Changing the container and the codecs (moving to theora and
> matroska) removes
> > > these audio drop issues, but this has other consequences such as
> needing more
> > > of the CPU resources to compress the video.
> > >
> > > Here is my gstreamer version:
> > > > gst-launch-1.0 --version
> > >
> > > gst-launch-1.0 version 1.10.4
> > > GStreamer 1.10.4
> > > [1]http://packages.qa.debian.org/gstreamer1.0
> > >
> > > Below are my experimental pipelines and the results observed.
> > >
> > > # Record C920 video and audio to a file
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
> video/x-h264,width=
> > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc
> device=2 !
> > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
> > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! filesink
> location='output.mp4'
> > > # Audio cuts out after 5 seconds and we also got "Can't record audio
> fast
> > > enough", no drop in CPU usage
> > >
> > > # Put the video audio recording infront of the video for recording
> C920 video
> > > and audio to a file
> > > gst-launch-1.0 -v pulsesrc device=2 ! queue ! audio/x-raw,format=S16LE,
> channels
> > > =2,rate=8000 ! queue ! audioconvert ! queue ! avenc_aac ! queue ! mux.
> v4l2src
> > > device=/dev/video1 ! queue ! video/x-h264,width=1920,
> height=1080,framerate=30/1
> > > ! queue ! h264parse ! queue ! mux. mpegtsmux name=mux ! filesink
> location=
> > > 'output.mp4'
> > > # Audio cuts out after 5 seconds and we also got "Can't record audio
> fast
> > > enough", no drop in CPU usage
> > >
> > > # Try a different mux for recording C920 video and audio to a file
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
> video/x-h264,width=
> > > 1920,height=1080,framerate=30/1 ! h264parse ! avdec_h264 ! theoraenc
> ! queue !
> > > mux. pulsesrc device=2 ! audio/x-raw,format=S16LE,channels=2,rate=8000
> !
> > > audioconvert ! queue ! avenc_aac ! queue ! mux. matroskamux name=mux !
> filesink
> > > location='output.mkv'
> > > # Audio and video was in sync for the whole video and we also got
> "Can't record
> > > audio fast enough", but video looked bad and CPU usage was huge
> > >
> > > # Try the mp3 audio codec
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
> video/x-h264,width=
> > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc
> device=2 !
> > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
> > > lamemp3enc ! queue ! mux. mpegtsmux name=mux ! filesink
> location='output.mp4'
> > > # Just flatout didn't start, perhaps I can only use AAC with the mp4
> container
> > >
> > > # Try a test src
> > > gst-launch-1.0 -v audiotestsrc ! queue ! audio/x-raw,format=S16LE,
> channels=
> > > 2,rate=8000 ! queue ! audioconvert ! queue ! avenc_aac ! queue ! mux.
> v4l2src
> > > device=/dev/video1 ! queue ! video/x-h264,width=1920,
> height=1080,framerate=30/1
> > > ! queue ! h264parse ! queue ! mux. mpegtsmux name=mux ! filesink
> location=
> > > 'output.mp4'
> > > # Didn't get any frame drop errors but audio still drops out after 5
> seconds
> > >
> > > # Try using a fake sink
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
> video/x-h264,width=
> > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc
> device=2 !
> > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
> > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! fakesink
> > > # We still got "Can't record audio fast enough"
> > >
> > > # Try putting a queue between mpegtsmux and filesink
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
> video/x-h264,width=
> > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc
> device=2 !
> > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! audioconvert ! queue !
> > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink
> location=
> > > 'output.mp4'
> > > # Audio drops out between 4 and 10 seconds and we also got "Can't
> record audio
> > > fast enough"
> > >
> > > # add sync=false at the sink
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-h264,width=1920,
> height=
> > > 1080,framerate=30/1 ! queue ! h264parse ! queue ! mux. pulsesrc
> device=2 !
> > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert
> ! queue !
> > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink
> location=
> > > 'output.mp4' sync=false
> > > # Audio cuts out at 4 seconds and we also got "Can't record audio fast
> enough"
> > >
> > > # Add provide-clock=false to pulsesrc from [2]
> https://stackoverflow.com/
> > > questions/35744114/getting-warning-messages-from-alsasrc
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
> video/x-h264,width=
> > > 1920,height=1080,framerate=30/1 ! h264parse ! queue ! mux. pulsesrc
> device=2
> > > provide-clock=false ! audio/x-raw,format=S16LE,channels=2,rate=8000 !
> > > audioconvert ! queue ! avenc_aac ! queue ! mux. mpegtsmux name=mux !
> queue !
> > > filesink location='output.mp4' sync=false
> > > # Doesn't help, no difference in performance Audio cuts out at 4
> seconds and we
> > > also got "Can't record audio fast enough"
> > >
> > > # Changed some queues around
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-h264,width=1920,
> height=
> > > 1080,framerate=30/1 ! queue ! h264parse ! queue ! mux. pulsesrc
> device=2 !
> > > audio/x-raw,format=S16LE,channels=2,rate=8000 ! queue ! audioconvert
> ! queue !
> > > avenc_aac ! queue ! mux. mpegtsmux name=mux ! queue ! filesink
> location=
> > > 'output.mp4' sync=false
> > > # Audio falls out after 5 seconds we also got "Can't record audio fast
> enough",
> > > no difference in performance
> > >
> > > # try using "voaacenc bitrate=65536 ! aacparse" from [3]https://
> > > stackoverflow.com/questions/23505166/
> > > use-gstreamer-to-stream-video-and-audio-of-logitech-c920
> > > gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-h264,width=1920,
> height=
> > > 1080,framerate=30/1 ! queue ! h264parse ! queue ! mux. pulsesrc
> device=2 !
> > > audio/x-raw ! queue ! audioconvert ! queue ! voaacenc bitrate=65536 !
> aacparse
> > > ! queue ! mux. mpegtsmux name=mux ! queue ! filesink
> location='output.mp4' sync
> > > =false
> > > # Audio falls out after 5 seconds we also got "Can't record audio fast
> enough",
> > > no difference in performance
> > >
> > > If you are intrigued by this issue please let me know your thoughts.
> > >
> > > References:
> > >
> > > [1] http://packages.qa.debian.org/gstreamer1.0
> > > [2] https://stackoverflow.com/questions/35744114/getting-
> warning-messages-from-alsasrc
> > > [3] https://stackoverflow.com/questions/23505166/use-
> gstreamer-to-stream-video-and-audio-of-logitech-c920
> > > _______________________________________________
> > > gstreamer-devel mailing list
> > > gstreamer-devel at lists.freedesktop.org
> > > https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> >
> >
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>

-- 
*W.A. Garrett Weaver *
weaverg at email.arizona.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180617/fe9c4ad0/attachment-0001.html>