Need guidance for implementing ice candidate pool

[Sherrill Lam]

Hi Olivier,

I have some follow up questions regarding your advice.

I've tried creating GstWebRTCICEStream objects ahead of time and using them
when gst_webrtc_ice_add_stream() was called. But when I tried to use these
pre-created GstWebRTCICEStream objects to gather ICE candidates, I couldn't
gather any reflexive candidates without gst_webrtc_ice_add_stream() being
called. According to Libnice logs, component_io_cb() was not triggered
until then.

I suspected it was because nicesrc element was not created thus
component_io_cb() was not triggered. So I tried pre-creating
TransportStream objects instead of GstWebRTCICEStream objects. And this
time, reflexive candidates could be pre-gathered before
gst_webrtc_ice_add_stream() being called. But I'm not sure if this is the
correct direction to go, as we would need to properly configure the
pre-created TransportStream objects so they could be used later (such as in
_get_or_create_data_channel_transports() and
_get_or_create_rtp_transport_channel()). Or maybe the
original GstWebRTCICEStream approach would work and I just missed
something. Any suggestions?

Also, I'm a bit confused about your suggestions with the
freezing/unfreezing candidates. When I read the RFC for frozen candidates
<https://datatracker.ietf.org/doc/html/rfc5245#section-2.4>, my
understanding is frozen candidates is a mechanism for candidate pairs, not
for individual candidates. In our scenarios, ice candidate pooling is only
used before SDP negotiation happens, which means there won't be any remote
candidates at the point and no candidate pairs are created yet. Could you
elaborate more on how the freezing/unfreezing candidates fit in the picture
here?

Thanks a lot!
Sherrill

On Sun, 9 May 2021 at 16:01, Sherrill Lam <sherrill.lam at gmail.com> wrote:

> Hi Olivier and Mathieu,
>
> Thanks a lot for your reply! When I think more more on it, I think Olivier
> is right that the implementation is more a webrtc level issue. I'm going to
> start from pre-allocating streams and caching ice candidates, and see how
> it goes.
>
> Thanks again for your great suggestions!
>
> Regards,
> Sherrill
>
> On Mon, 3 May 2021 at 15:54, Olivier Crête via gstreamer-devel <
> gstreamer-devel at lists.freedesktop.org> wrote:
>
>> Hi,
>>
>> Giving this a bit more though, I guess one could create those unallocated
>> streams hidden inside libnice and expose them when nice_agent_add_stream()
>> is called, but it's a bit trickier as that means forcing eveyr stream to
>> have the same number of components, which isn't standard in ICE (but it's
>> always true for webrtc as rtcp-mux is mandated now).
>>
>> Olivier
>>
>> On Mon, 2021-05-03 at 15:42 -0400, Olivier Crête via gstreamer-devel
>> wrote:
>>
>> Hi,
>>
>> With my libnice maintainer hat on, it's definitely something I would
>> accept.
>>
>> The tricky part is the implementation. At first, I though we'd want to
>> create a new object separate from the ICE agent to do the gathering, but it
>> gets quite tricky quickly as the gathering and especially TURN code is
>> quite tied to the ICE agent code.
>>
>> And looking at the spec, I think the implementation is mostly a webrtc
>> level issue. All that you need to do is to add a new API to GstWebRTCICE to
>> create GstWebRTCICEStream objects ahead of time and "cache" them, then when
>> gst_webrtc_ice_add_stream() is called, you can just return one of the
>> pre-allocated streams. The only issue is that we need to add an API to
>> libnice to mark a stream "unallocated" so that it gets ignored when doing
>> the freezing/unfreezing dance, and the same thing needs to happen inside
>> webrtcbin so they get ignored when trying to compute the overall agent
>> state.
>>
>> Olivier
>>
>>
>> On Mon, 2021-05-03 at 21:21 +0200, Mathieu Duponchelle via
>> gstreamer-devel wrote:
>>
>> Hey, this seems interesting, perhaps open an issue in libnice's gitlab to
>> discuss this?
>>
>> On 5/2/21 5:29 AM, Sherrill Lam via gstreamer-devel wrote:
>>
>> Hi there!
>>
>> My team is interested to implement a prefetch ice candidate pool and add
>> an ice-candidate-pool-size property ( [rtc spec](
>> https://www.w3.org/TR/webrtc/#dom-rtcconfiguration-icecandidatepoolsize)
>> ) for webrtcbin element. We think it will help accelerate the SDP
>> generation at startup process. We’d like to get more guidance on what would
>> be the best way to do it.
>>
>> Although ice candidate pool is a webrtc concept, I think it might make
>> more sense if the pool itself is implemented within NiceAgent in Libnice.
>> We could add an API in Libnice to create such a pool with a specified pool
>> size and start generating ice candidates. When
>> nice_agent_gather_candidates() is called, it will first check if there are
>> candidates in the pool before generating new ones.
>>
>> Any thoughts or suggestions?
>>
>> Thanks!
>> Sherrill
>>
>> _______________________________________________
>>
>> gstreamer-devel mailing list
>>
>> gstreamer-devel at lists.freedesktop.org
>>
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>> --
>>
>> Olivier Crête
>> olivier.crete at collabora.com
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210524/70b8058b/attachment.htm>