Adventures with GStreamer and the C920

[W.A. Garrett Weaver]

On June 17th, with the help of this mailing list, I was able to figure out
how to record video and audio off my C920 webcam, sync them up and write
the streams to an mp4 file.

This is the pipeline below.
gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-h264,width=1920,height=1080,framerate=30/1 ! h264parse ! queue
max-size-buffers=0 max-size-bytes=0 max-size-time=1000000000 ! mux.
pulsesrc device=1 ! audioconvert ! queue ! avenc_aac ! queue ! mux.
mpegtsmux name=mux ! filesink location='output.mp4'

You'll notice that video is pulled from the webcam as video/x-h264 because
the webcam has a built-in h264 encoder. I got this webcam specifically for
that feature. I didn't want my poor little laptop straddled with encoding
raw 1080p video into h264 on its own.

What makes this pipeline work is the `queue max-size-buffers=0
max-size-bytes=0 max-size-time=1000000000' otherwise gstreamer reports that
audio samples are dropped because downstream can't keep up. This results in
no audio being in the video. I couldn't have figured out how to fix that
without this mailing list, so thank you all for the time you spend here
helping others.

The pipeline worked just fine when I tested it in my room. I made this
script in anticipation of filming the start of a web series with my
roommate. I was very excited to demonstrate the power of GNU/Linux for
working with multi-media because my roommate once told me "Linux is just
code." For some reason I felt the need to disprove that notion.

Last night we were all ready to film our web-series. We set up on a table
in the living room and faced the webcam towards a wall. I did my background
research for the video, my co-host put her makeup on, we were set. Then I
ran the pipeline, we were rolling, and then it happened. The dreaded
message appeared "Dropped 11907 samples. This is most likely because
downstream can't keep up and is consuming samples too slowly." Just like
before I added `queue max-size-buffers=0 max-size-bytes=0
max-size-time=1000000000'.

I stopped recording the video. I couldn't understand how this happened
again. What had changed? I opened up the script, tried extending queue
buffer sizes in a desperate attempt to get the pipeline working. My co-host
yelled "Don't try to solve technical issues, that takes hours!" She pushed
my laptop to the edge of the table and replaced it with a stack of books to
place her phone on.

We filmed the first episode of the webseries on my roommates iPhone.

I spent the afternoon today trying to reproduce the bug in my room to no
avail. I wondered if the bug was processor dependent, even though my laptop
wasn't using any more than 30% handling the pipeline. I tried unplugging my
laptop from the wall, I tried opening up the same tabs in chromium I had
open last night and could not reproduce the issue. The recording worked
fine today. Why did it fail when I needed it the most? Perhaps it was a
fluke?

I knew there had to be some way to reproduce the bug. I decided to
reproduce every condition that caused the issue, even moving my laptop in
the same place where the issue had happened. I ran the pipeline and then I
finally got the dreaded downstream can't keep up message again. Weird I
thought. I changed every variable but location, even rebooting my laptop,
and I still got the downstream issue with my laptop in the living room.
Then I moved my laptop back to my room. Re-ran the pipeline and the whole
thing ran beautifully. No issues.

What the heck?

Moving my laptop ten feet made the difference between the pipeline working
and the pipeline not working. Could it be that the table was hotter? Was my
neighbor closest to the living room operating a nuclear reactor? Was the
living room positioned over an Indian burial ground?

I thought the location issue was merely coincidental. I was just happy that
I could reproduce the bug in the first place. I thought the pipeline wasn't
dependable enough and if I could improve it in its worst state, it would
work 100% of the time. I tried for hours playing around with different
pipeline elements and settings to get recording working optimally. This
work was fruitless. I could not get the downstream bug to go away.

Then I started moving the webcam around and pointed it to the kitchen. I
recorded with the same old pipeline, it all worked, no downstream can't
keep up messages, then I turned the webcam 180 degrees around, back to me
and the wall behind me and then the downstream can't keep up messages
showed up again. Ah Ha!

I don't know why, but for some reason if all the background is too close
and has a matte color, the audio stream in the pipeline can't keep up.
Perhaps the background consistency changes the latency of the h264 encoder
on the C920 webcam. I have no idea, but this doesn't seem like something I
can debug.

Fortunately I did some research and found that GStreamer has built in
hardware accelerated video encoding with the vaapih264enc element and my
laptop has support for this element. I made a new pipeline that takes raw
1080p video off the webcam and compresses it using vaapih264enc.

gst-launch-1.0 -v v4l2src device=/dev/video1 ! queue !
video/x-raw,width=1920,height=1080,framerate=30/1 ! vaapih264enc !
h264parse ! queue max-size-buffers=0 max-size-bytes=0
max-size-time=1000000000 ! mux. pulsesrc device=1 ! audioconvert ! queue !
avenc_aac ! queue ! mux. mpegtsmux name=mux ! filesink location='output.mp4'

That pipeline works beautifully. It has consistent performance regardless
of location. It's great! I'm all ready to record the next episode. I'm
still curious why the first pipeline didn't work depending on the
background and I'm curious to know your thoughts for why that is.

-- 
*W.A. Garrett Weaver *
weaverg at email.arizona.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20180701/8d3b95bd/attachment-0001.html>