Can't make webrtcbin connect to H.264 offering clients

[Neil Young]

I'm hanging on this for a week now, and it seems so easy, but whatever I do, I can't make it work

My app is a Python GST app, running on 18.4. It supports two modes: "call" and "answer". In call mode the app immediately issues an H.264 or VP8 OFFER (depending on a setting). This works in 100 % of the cases. 

What also works is when the app ANSWERs a VP8 offer. Just H.264 OFFERS are not working, even though the SDP answer looks good. I just don't get the video to remote.

My app works like so:

1) When the app has a connection to the signaling server it creates a webrtcbin object:

    def build_webrtcbin(self):
        self.webrtcbin = Gst.ElementFactory.make("webrtcbin")
        self.webrtcbin.set_property("bundle-policy", "max-bundle")
        self.webrtcbin.set_property("stun-server", "stun://stun.l.google.com:19302")
        self.webrtcbin.connect('on-negotiation-needed', self.on_negotiation_needed)
        self.webrtcbin.connect('on-ice-candidate', self.on_ice_candidate)
        self.webrtcbin.connect('pad-added', self.on_pad_added)
        self.webrtcbin.connect('notify::connection-state', self.on_connection_state)
        self.webrtcbin.connect('notify::signaling-state', self.on_signaling_state)
        self.webrtcbin.connect('notify::ice-connection-state', self.on_ice_connection_state)
        self.webrtcbin.connect('notify::ice-gathering-state', self.on_ice_gathering_state)

        self.pipeline = Gst.Pipeline.new()
        self.pipeline.add(self.webrtcbin)

Nothing special I guess.

2) When the SDP OFFER comes in, the app does this:

2a) It starts the H.264 pipeline with `start_pipeline`. 
2b) It emits the remote description to the local webrtcbin

self.webrtcbin.emit('set-remote-description', desc, promise)

2c) It creates an answer calling `self.create_answer()`

    def create_answer(self):
        promise = Gst.Promise.new_with_change_func(self.on_answer_created, self.webrtcbin, None)
        self.webrtcbin.emit('create-answer', None, promise)
    
    def on_answer_created(self, promise, _, __):
        ''' Local WebRTC stack has created an answer '''
        promise.wait()
        reply = promise.get_reply()
        answer = reply.get_value('answer')
        promise = Gst.Promise.new()
        self.webrtcbin.emit('set-local-description', answer, promise)
        promise.interrupt()
        self.send_sdp_answer(answer) # Forwarding the answer to remote using the signaling server

Observations so far:

- It doesn't work if I don't set a payload-type which matches a payload type carried by the OFFERER or if I omit the payload-type. With ALL browsers it is sufficient to use the payload-type carried by the first H.264 element in the OFFER
- It is not necessary to set the `profile-level-id`

The best result I can achieve with this flow is a spinning wheel in the browser. No video.

BUT: 

When I issue a call to `start_pipeline` somewhere shortly after the start (I use the moment, when I have a connection with the signaling server for this) with _exactly_the_same_payload_type_that_will_match_later_, I 100% have a video...

This MUST be a side effect, because the log complains about the second `start_pipeline` (warnings, it seems to ignore it).... But it works. Of course this is not a working solution, because

a) I would have to know the payload-type beforehand (it doesn't work if I use another payload-type in the first start)
b) I would have to parse the incoming SDP for the first H.264 (which is doable)

What am I doing wrong?




    def start_pipeline(self, payload_type):
        videotestsrc = Gst.ElementFactory.make("videotestsrc", "videotestsrc")
        videotestsrc.set_property("is-live", True)
        videotestsrc.set_property("pattern", "smpte")

        videotestsrc_caps = Gst.caps_from_string("video/x-raw, width=1280, height=720")
      
        videotestsrc_caps_filter = Gst.ElementFactory.make("capsfilter")
        videotestsrc_caps_filter.set_property("caps", videotestsrc_caps)

        videoconvert = Gst.ElementFactory.make("videoconvert", "videoconvert")

        x264enc = Gst.ElementFactory.make("x264enc", "x264enc")
        x264enc.set_property("threads", 4)
    
        x264enc_caps = Gst.caps_from_string("video/x-h264, profile=constrained-baseline")

        x264enc_caps_filter = Gst.ElementFactory.make("capsfilter")
        x264enc_caps_filter.set_property("caps", x264enc_caps)

        rtph264pay = Gst.ElementFactory.make("rtph264pay", "rtph264pay")
        rtph264pay.set_property("config-interval", -1)
        rtph264pay.set_property("aggregate-mode", "zero-latency")

        rtph264pay_caps = Gst.caps_from_string("application/x-rtp, media=video, encoding-name=H264")
        rtph264pay_caps.set_value("payload", payload_type)

        rtph264pay_caps_filter = Gst.ElementFactory.make("capsfilter")
        rtph264pay_caps_filter.set_property("caps", rtph264pay_caps)

        self.pipeline.add(videotestsrc)
        self.pipeline.add(videotestsrc_caps_filter)
        self.pipeline.add(videoconvert)
        self.pipeline.add(x264enc)
        self.pipeline.add(x264enc_caps_filter)
        self.pipeline.add(rtph264pay)
        self.pipeline.add(rtph264pay_caps_filter)

        Gst.Element.link(videotestsrc, videotestsrc_caps_filter)
        Gst.Element.link(videotestsrc_caps_filter, videoconvert)
        Gst.Element.link(videoconvert, x264enc)
        Gst.Element.link(x264enc, x264enc_caps_filter)
        Gst.Element.link(x264enc_caps_filter, rtph264pay)
        Gst.Element.link(rtph264pay, rtph264pay_caps_filter)
        Gst.Element.link(rtph264pay_caps_filter, self.webrtcbin)
        self.pipeline.set_state(Gst.State.PLAYING)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210821/3ded4653/attachment-0001.htm>