[Pipewire-devel] Query regarding h264 format on Raspberry Pi when Pipewire is being used with libcamera

[Aditya Kulkarni]

Hello George and Wim,

Please guide us on the queries below as well, in your reply.

1. Is it possible to enable libcamera spa plugin and v4l2 spa plugin
simultaneously?
2. Is it possible to achieve communication between two spa-plugins, in
pipewire?
3. In the 3rd approach which you mentioned, i.e., "Use PipeWire to capture
YUV from libcamera, then a GStreamer app to encode and make the stream
available again in the PipeWire graph, then a separate app to stream". In
this scenario, can pipewire graph handle, in case the encoded stream is
requested by multiple clients at a time?

Looking forward to your reply.

Thanks & Regards,
Aditya

On Wed, Oct 11, 2023 at 3:11 PM Aditya Kulkarni <avkulkarni304 at gmail.com>
wrote:

> Hello George,
>
> Our understanding is, in the spa v4l2 plugin we have
> SPA_MEDIA_SUBTYPE_h264 as SPA_TYPE_OBJECT_Format through which we can
> get/request h264 format from V4L2.
>
> I'd request you to please give some insight whether the understanding is
> correct because you mentioned PipeWire’s v4l2 spa plugin only supports
> accessing video capture devices such as cameras, *with raw format only*.
>
> Thanks & Regards,
> Aditya
>
> On Tue, Oct 10, 2023 at 1:08 PM George Kiagiadakis <
> george.kiagiadakis at collabora.com> wrote:
>
>> Hello Aditya,
>>
>> PipeWire’s v4l2 spa plugin does not support hardware memory-to-memory
>> devices (encoders, decoders, etc). It only supports accessing video capture
>> devices such as cameras, with raw format only. We do not have any plans to
>> extend this plugin to work with m2m devices, as there’s a lot of complexity
>> around this functionality and very little gain from using PipeWire for this
>> job instead of a more full-fledged multimedia framework, such as GStreamer.
>>
>> Using GStreamer, this could be made so much easier. I can think of 3
>> different ways of doing it:
>>
>>
>> 1. Use PipeWire to capture YUV from libcamera, then use a GStreamer
>> application to encode and stream.
>>
>> PipeWire [spa libcamera] —> Application [ pipewiresrc ! v4l2encoder !
>> streaming ]
>>
>>
>> 2. Use GStreamer to do everything, remove PipeWire.
>>
>> Application [ libcamerasrc ! v4l2encoder ! streaming ]
>>
>>
>> 3. Use PipeWire to capture YUV from libcamera, then a GStreamer app to
>> encode and make the stream available again in the PipeWire graph, then a
>> separate app to stream.
>>
>> PipeWire [spa libcamera] —> Application [ pipewiresrc ! v4l2encoder !
>> pipewiresink ] —> Application [ pipwiresrc ! streaming ]
>>
>> This last option is the most complex one and only makes sense in some
>> advanced scenarios, such as when the streaming application is containerized
>> and you don’t want to give it access to the hardware encoder. Also, note
>> that this option has never been tried, afaik, and there may be rough edges
>> related to passing encoded video content through PipeWIre. But in theory it
>> should be possible and any bugs can be fixed eventually.
>>
>> I think that choosing one of these 3 options is a matter of understanding
>> what PipeWire can offer in options 1 and 3 and considering what is your
>> actual use case and goals.
>>
>> I hope this helps.
>>
>> Best regards,
>> George
>>
>>
>> On 9 Oct 2023, at 16:39, Aditya Kulkarni <avkulkarni304 at gmail.com> wrote:
>>
>> Hello Wim,
>>
>> Thanks for your reply. We checked with the libcamera team on adding h264
>> support but they confirmed that the addition of h264 support in libcamera
>> is out of libcamera's scope.
>>
>> We are trying to find a way to achieve h264 using pipewire for our live
>> streaming use case.
>>
>> We can get encoded data from the R-Pi which has hardware encoder but
>> libcamera has support to extract raw format.
>>
>> *What we thought is, we can get the raw format by using the libcamera spa
>> plugin of pipewire and this raw format can be passed to v4l2 spa plugin of
>> pipewire for encoding. Further, v4l2 spa plugin in pipewire will interact
>> with v4l2 driver to get h264 format from R-Pi hardware encoder.*
>>
>> The High level flow diagram of the above approach has been attached to
>> this email for your reference.
>>
>> The above approach demands the usage of two spa plugins simultaneously.
>>
>>
>>
>> *In this scenario,1. Is it possible to enable libcamera spa plugin and
>> v4l2 spa plugin at the same time? 2. Is it possible to achieve
>> inter-spa-plugin communication, in pipewire?*
>>
>> Can you please guide us on this.
>>
>> We would also like to know from you, in case, there is any better
>> approach to achieve this.
>>
>> Thanks and Regards,
>> Aditya
>>
>> On Thu, Aug 31, 2023 at 7:19 PM Wim Taymans <wim.taymans at gmail.com>
>> wrote:
>>
>>> On Thu, 31 Aug 2023 at 15:38, Aditya Kulkarni <avkulkarni304 at gmail.com>
>>> wrote:
>>> >
>>> > Hello,
>>> >
>>> > We are looking for Camera live streaming using h264 format on
>>> Raspberry Pi.
>>> >
>>> > We are using the Pipewire framework to interact with applications and
>>> libcamera.
>>> >
>>> > But, we are facing pw.link: (39.0 -> 47.0) negotiating -> error (no
>>> more input formats).
>>> >
>>> > As libcamera doesn't have H264 support, can you please guide on how to
>>> achieve H264 using pipewire, when libcamera is used.
>>>
>>> It is not possible, libcamera does not support h264.
>>>
>>> Wim
>>>
>>> >
>>> > Hoping for your reply.
>>> >
>>> > Thanks & Regards,
>>> > Aditya
>>> >
>>> >
>>> >
>>>
>> <High_level_approach_diagram.png>
>>
>>
>>
>> --
>> George Kiagiadakis
>> Principal Software Engineer
>>
>> Collabora Ltd.
>> Platinum Building, St John's Innovation Park, Cambridge CB4 0DS, UK
>> Registered in England & Wales, no. 5513718
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/pipewire-devel/attachments/20231013/dc7140ed/attachment.htm>