libcamerasrc and formats?

[Nicolas Dufresne]

Hi,

Le mercredi 22 mars 2023 à 10:15 +0100, Florian Echtler a écrit :
> P.S. a follow-up question: at the moment, I can't use libcamerasrc to directly 
> stream H.264 from the camera, correct?

There is issues, but no roadblock. I made a recorder the otherday, here's what
the pipeline looks like:

pipeline = Gst.parse_launch(
    f"libcamerasrc ! identity name=eos-dummy ! video/x-
raw,format={args.format},width={args.width},height={args.height},framerate={args
.framerate}/1 ! " +
    f"capssetter caps=\"video/x-raw,interlace-
mode=(string)progressive,colorimetry=(string)bt601\" " + 
    f"! v4l2h264enc output-io-mode={encoder_io_mode} extra-
controls=s,video_bitrate={args.bitrate} ! video/x-h264,profile=constrained-
baseline,level=(string)4 ! h264parse "
    f"! splitmuxsink max-size-time={args.segment * 1000000}
location=video%02d.mov"


And here's the list of issues we are working around:

1. eos-dummy, just something with static pads to push eos, as this is broken in
libcamerasrc
2. capssetter, works around v4l2videodec issues around colorimetry, and
libcamerasrc bugs of pushing incomplete video/x-raw caps
3. video/x-h264,pr ... the encoder get configure with invalid profile/level and
fail to streamon otherwise
4. output-io-mode=, dmabuf-import will enable zero-copy, but only works for
YUY2, we have respectively 2 and 3 dmabufs from libcamerasrc in NV12 and I420,
but the encoder only accepts 1 dmabuf.
5. You must select a pixel format, libcamerasrc defaults to NV21 which is broken
in encoder firmware or driver (still have to properly report that).

But with all these workaround, the encoder works fine, and you can replace
splitmuxsink with some rtpsink, or wrap this in an rtsp server. I'm haven't
tried the new webrtcsink element though, but in theory it should be possible.

Nicolas

> 
> On 17.03.23 20:21, Nicolas Dufresne wrote:
> > Hi,
> > 
> > Le vendredi 17 mars 2023 à 10:21 +0100, Florian Echtler a écrit :
> > > Thank you - turns out it's rather hard to build libcamera on the target device,
> > > which is a Raspberry Pi Zero. I'll test with another Raspi ASAP.
> > > 
> > 
> > Meson based projects aren't too hard to cross compile, there is a concises
> > instruction how to set this up a toolchain for RPi OS:
> > 
> > https://deardevices.com/2019/04/18/how-to-crosscompile-raspi/
> > 
> > I would have hoped for already made docker image. I never tried this. An
> > alternative I've used (though I have a Pi4 now with 8G of RAM), is to use qemu
> > to chroot into the SD card rootfs and just build using emulation.
> > 
> > regards,
> > Nicolas
> > 
> > > 
> > > Best, Florian
> > > 
> > > On 15.03.23 19:13, Nicolas Dufresne wrote:
> > > > Hi,
> > > > 
> > > > Le mercredi 15 mars 2023 à 17:24 +0100, Florian Echtler via gstreamer-devel a
> > > > écrit :
> > > > > Hello everyone,
> > > > > 
> > > > > I'm experimenting with a Raspberry Pi camera and would like to process the
> > > > > frames with OpenCV. Apparently, the "modern" way of doing so is through
> > > > > libcamera, and for OpenCV, this means using a GStreamer pipeline with libcamerasrc.
> > > > > 
> > > > > For efficiency reasons, I'd like to get a grayscale image directly from the
> > > > > camera to avoid unneeded conversions; so I used the following pipeline:
> > > > > 
> > > > > libcamerasrc !
> > > > > video/x-raw,width=(int)640,height=(int)480,framerate=(fraction)30/1,format=GRAY
> > > > > ! appsink
> > > > > 
> > > > > "GRAY" is listed as a supported format for the camera in the Video4Linux device,
> > > > > but I don't get a working pipeline when I try that. If I leave out the format
> > > > > parameter, then I'm getting NV12 (or I can pick something generic like RGB). How
> > > > > can I see which formats I can actually get out of libcamerasrc and the sensor?
> > > > 
> > > > Any chance you can give this patch a try ? I'll post it to the libcamera project
> > > > if you can confirm its working. Note that libcamerasrc is entirely developed as
> > > > part of libcamera project.
> > > > 
> > > > ---
> > > > commit abf017f5c0f547ceaa863e658f22af9f547a863d (HEAD -> logger-crash)
> > > > Author: Nicolas Dufresne <nicolas.dufresne at collabora.com>
> > > > Date:   Wed Mar 15 14:10:08 2023 -0400
> > > > 
> > > >       gstreamer: Map R8 pixel format
> > > >       
> > > >       This enables monochrome 8bit to be supported in libcamerasrc.
> > > >       
> > > >       Signed-off-by: Nicolas Dufresne <nicolas.dufresne at collabora.com>
> > > > 
> > > > diff --git a/src/gstreamer/gstlibcamera-utils.cpp b/src/gstreamer/gstlibcamera-
> > > > utils.cpp
> > > > index 750ec351..f1e34b60 100644
> > > > --- a/src/gstreamer/gstlibcamera-utils.cpp
> > > > +++ b/src/gstreamer/gstlibcamera-utils.cpp
> > > > @@ -26,6 +26,9 @@ static struct {
> > > >           { GST_VIDEO_FORMAT_ENCODED, formats::SGRBG8 },
> > > >           { GST_VIDEO_FORMAT_ENCODED, formats::SRGGB8 },
> > > >    
> > > > +       /* Monochrome */
> > > > +       { GST_VIDEO_FORMAT_GRAY8, formats::R8 },
> > > > +
> > > >           /* RGB16 */
> > > >           { GST_VIDEO_FORMAT_RGB16, formats::RGB565 },
> > > >    
> > > > 
> > > > 
> > > > 
> > > 
> > > -- 
> > > SENT FROM MY DEC VT50 TERMINAL
> > > 
> > 
> 
> -- 
> SENT FROM MY DEC VT50 TERMINAL
>