[Pipewire-devel] Query regarding h264 format on Raspberry Pi when Pipewire is being used with libcamera

[Aditya Kulkarni]

Hello George,

Thanks for your reply.

Is pipewire capable of supporting cameras generating multiple encoded
streams?

Is it possible to have a call with your team, as per your convenience?

Thanks & Regards,
Aditya

On Fri, Oct 13, 2023 at 9:36 PM George Kiagiadakis <
george.kiagiadakis at collabora.com> wrote:

> Hi Aditya,
>
> [replies inline]
>
> On 13 Oct 2023, at 07:42, Aditya Kulkarni <avkulkarni304 at gmail.com> wrote:
>
> Hello George and Wim,
>
> Please guide us on the queries below as well, in your reply.
>
> 1. Is it possible to enable libcamera spa plugin and v4l2 spa plugin
> simultaneously?
>
>
> Yes, both are actually enabled by default.
>
> 2. Is it possible to achieve communication between two spa-plugins, in
> pipewire?
>
>
> Yes, you can set them up as separate nodes in the graph and then link them
> as usual. This is the same as linking such a node to an application.
>
> 3. In the 3rd approach which you mentioned, i.e., "Use PipeWire to capture
> YUV from libcamera, then a GStreamer app to encode and make the stream
> available again in the PipeWire graph, then a separate app to stream". In
> this scenario, can pipewire graph handle, in case the encoded stream is
> requested by multiple clients at a time?
>
>
> Yes, it should be possible to capture this from multiple clients.
>
> Looking forward to your reply.
>
> Thanks & Regards,
> Aditya
>
> On Wed, Oct 11, 2023 at 3:11 PM Aditya Kulkarni <avkulkarni304 at gmail.com>
> wrote:
>
>> Hello George,
>>
>> Our understanding is, in the spa v4l2 plugin we have
>> SPA_MEDIA_SUBTYPE_h264 as SPA_TYPE_OBJECT_Format through which we can
>> get/request h264 format from V4L2.
>>
>> I'd request you to please give some insight whether the understanding is
>> correct because you mentioned PipeWire’s v4l2 spa plugin only supports
>> accessing video capture devices such as cameras, *with raw format only*.
>>
>>
> Indeed, the h264 format is advertised. I may be wrong on saying it
> supports raw only. It can probably support anything that the camera
> supports. I am not sure if anyone has ever tested this, though.
>
>
> Thanks & Regards,
>> Aditya
>>
>> On Tue, Oct 10, 2023 at 1:08 PM George Kiagiadakis <
>> george.kiagiadakis at collabora.com> wrote:
>>
>>> Hello Aditya,
>>>
>>> PipeWire’s v4l2 spa plugin does not support hardware memory-to-memory
>>> devices (encoders, decoders, etc). It only supports accessing video capture
>>> devices such as cameras, with raw format only. We do not have any plans to
>>> extend this plugin to work with m2m devices, as there’s a lot of complexity
>>> around this functionality and very little gain from using PipeWire for this
>>> job instead of a more full-fledged multimedia framework, such as GStreamer.
>>>
>>> Using GStreamer, this could be made so much easier. I can think of 3
>>> different ways of doing it:
>>>
>>>
>>> 1. Use PipeWire to capture YUV from libcamera, then use a GStreamer
>>> application to encode and stream.
>>>
>>> PipeWire [spa libcamera] —> Application [ pipewiresrc ! v4l2encoder !
>>> streaming ]
>>>
>>>
>>> 2. Use GStreamer to do everything, remove PipeWire.
>>>
>>> Application [ libcamerasrc ! v4l2encoder ! streaming ]
>>>
>>>
>>> 3. Use PipeWire to capture YUV from libcamera, then a GStreamer app to
>>> encode and make the stream available again in the PipeWire graph, then a
>>> separate app to stream.
>>>
>>> PipeWire [spa libcamera] —> Application [ pipewiresrc ! v4l2encoder !
>>> pipewiresink ] —> Application [ pipwiresrc ! streaming ]
>>>
>>> This last option is the most complex one and only makes sense in some
>>> advanced scenarios, such as when the streaming application is containerized
>>> and you don’t want to give it access to the hardware encoder. Also, note
>>> that this option has never been tried, afaik, and there may be rough edges
>>> related to passing encoded video content through PipeWIre. But in theory it
>>> should be possible and any bugs can be fixed eventually.
>>>
>>> I think that choosing one of these 3 options is a matter of
>>> understanding what PipeWire can offer in options 1 and 3 and considering
>>> what is your actual use case and goals.
>>>
>>> I hope this helps.
>>>
>>> Best regards,
>>> George
>>>
>>>
>>> On 9 Oct 2023, at 16:39, Aditya Kulkarni <avkulkarni304 at gmail.com>
>>> wrote:
>>>
>>> Hello Wim,
>>>
>>> Thanks for your reply. We checked with the libcamera team on adding h264
>>> support but they confirmed that the addition of h264 support in libcamera
>>> is out of libcamera's scope.
>>>
>>> We are trying to find a way to achieve h264 using pipewire for our live
>>> streaming use case.
>>>
>>> We can get encoded data from the R-Pi which has hardware encoder but
>>> libcamera has support to extract raw format.
>>>
>>> *What we thought is, we can get the raw format by using the libcamera
>>> spa plugin of pipewire and this raw format can be passed to v4l2 spa plugin
>>> of pipewire for encoding. Further, v4l2 spa plugin in pipewire will
>>> interact with v4l2 driver to get h264 format from R-Pi hardware encoder.*
>>>
>>> The High level flow diagram of the above approach has been attached to
>>> this email for your reference.
>>>
>>> The above approach demands the usage of two spa plugins simultaneously.
>>>
>>>
>>>
>>> *In this scenario,1. Is it possible to enable libcamera spa plugin and
>>> v4l2 spa plugin at the same time? 2. Is it possible to achieve
>>> inter-spa-plugin communication, in pipewire?*
>>>
>>> Can you please guide us on this.
>>>
>>> We would also like to know from you, in case, there is any better
>>> approach to achieve this.
>>>
>>> Thanks and Regards,
>>> Aditya
>>>
>>> On Thu, Aug 31, 2023 at 7:19 PM Wim Taymans <wim.taymans at gmail.com>
>>> wrote:
>>>
>>>> On Thu, 31 Aug 2023 at 15:38, Aditya Kulkarni <avkulkarni304 at gmail.com>
>>>> wrote:
>>>> >
>>>> > Hello,
>>>> >
>>>> > We are looking for Camera live streaming using h264 format on
>>>> Raspberry Pi.
>>>> >
>>>> > We are using the Pipewire framework to interact with applications and
>>>> libcamera.
>>>> >
>>>> > But, we are facing pw.link: (39.0 -> 47.0) negotiating -> error (no
>>>> more input formats).
>>>> >
>>>> > As libcamera doesn't have H264 support, can you please guide on how
>>>> to achieve H264 using pipewire, when libcamera is used.
>>>>
>>>> It is not possible, libcamera does not support h264.
>>>>
>>>> Wim
>>>>
>>>> >
>>>> > Hoping for your reply.
>>>> >
>>>> > Thanks & Regards,
>>>> > Aditya
>>>> >
>>>> >
>>>> >
>>>>
>>> <High_level_approach_diagram.png>
>>>
>>>
>>>
>>> --
>>> George Kiagiadakis
>>> Principal Software Engineer
>>>
>>> Collabora Ltd.
>>> Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
>>> Registered in England & Wales, no. 5513718
>>>
>>>
> --
> George Kiagiadakis
> Principal Software Engineer
>
> Collabora Ltd.
> Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
> Registered in England & Wales, no. 5513718
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/pipewire-devel/attachments/20231016/ac5dedc4/attachment.htm>