Can't make webrtcbin connect to H.264 offering clients
Neil Young
foreverneilyoung at googlemail.com
Mon Aug 23 04:09:12 UTC 2021
I'm sure you won't publish this again. But before I go, this much: The entire support around GStreamer/Webrtcbin is so peppered with arrogant rebukes, "don't post here, go somewhere else" rejections, with half-truths, only designed to have as little as possible to do with the problems of the other, that it just stinks. Not even beginning to assume that you might have a problem, even when ways to investigate the problem are more or less laid at your feet. "No, we don't have a problem there, it's all your goofiness, move on".... If necessary, you are getting blocked because you are just annoying. I regret all the time I spent with you in nonsensical lists like this, prepared reproducible code samples and started discussions, or on your issue tracker... Fortunately, there are alternatives, and if I just lift a Chromium with Node/Puppeteer, then I know at the end in any case that it works and I don't have to bother with you anymore.
Webrtc is fortunately bigger than you guys, even if you think, you are the greatest.
> Am 22.08.2021 um 20:54 schrieb Neil Young <foreverneilyoung at googlemail.com>:
>
> OK, as I thought...
>
> In order to bring the discussion back from the land of legends to the real situations, these are the facts:
>
> My Chrome browser sends this "m" line:
>
> m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 121 127 120 125 107 108 109 35 36 124 119 123 118 114 115 116
>
> From the offered video codecs, these are the specifications for the H.264 codecs:
>
> a=rtpmap:102 H264/90000
> a=rtcp-fb:102 goog-remb
> a=rtcp-fb:102 transport-cc
> a=rtcp-fb:102 ccm fir
> a=rtcp-fb:102 nack
> a=rtcp-fb:102 nack pli
> a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
>
> a=rtpmap:108 H264/90000
> a=rtcp-fb:108 goog-remb
> a=rtcp-fb:108 transport-cc
> a=rtcp-fb:108 ccm fir
> a=rtcp-fb:108 nack
> a=rtcp-fb:108 nack pli
> a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
>
> a=rtpmap:123 H264/90000
> a=rtcp-fb:123 goog-remb
> a=rtcp-fb:123 transport-cc
> a=rtcp-fb:123 ccm fir
> a=rtcp-fb:123 nack
> a=rtcp-fb:123 nack pli
> a=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032
>
> a=rtpmap:124 H264/90000
> a=rtcp-fb:124 goog-remb
> a=rtcp-fb:124 transport-cc
> a=rtcp-fb:124 ccm fir
> a=rtcp-fb:124 nack
> a=rtcp-fb:124 nack pli
> a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0032
>
> a=rtpmap:125 H264/90000
> a=rtcp-fb:125 goog-remb
> a=rtcp-fb:125 transport-cc
> a=rtcp-fb:125 ccm fir
> a=rtcp-fb:125 nack
> a=rtcp-fb:125 nack pli
> a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
>
> a=rtpmap:127 H264/90000
> a=rtcp-fb:127 goog-remb
> a=rtcp-fb:127 transport-cc
> a=rtcp-fb:127 ccm fir
> a=rtcp-fb:127 nack
> a=rtcp-fb:127 nack pli
> a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
>
>
> Depending on what I set as "profile-level-id" in my started pipeline (see "start_pipeline" below) I really can make webrtcbin generate a seemingly matching ANSWER. For instance if I set this:
>
> rtph264pay_caps = Gst.caps_from_string("application/x-rtp, media=video, encoding-name=H264, profile-level-id=(string)42001f")
>
> the answer is:
>
> a=rtpmap:102 H264/90000
> a=rtcp-fb:102 nack pli
> a=rtcp-fb:102 ccm fir
> a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
> a=sendrecv
>
>
> or with
>
> rtph264pay_caps = Gst.caps_from_string("application/x-rtp, media=video, encoding-name=H264, profile-level-id=(string)42e01f")
>
> the answer is:
>
> a=rtpmap:125 H264/90000
> a=rtcp-fb:125 nack pli
> a=rtcp-fb:125 ccm fir
> a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
> a=sendrecv
>
> This all looks pretty good. And if I use my camera instead of "videotestsrc" I can see the camera light going on...
>
> I can also play with different packetization-modes --- each time the answer is matching...
>
> But even with these positive signs, all I get on the remote side is a spinning wheel in a black window...
>
> I can only get video if I start with the proper payload set TWICE... As described... This drives me crazy...
>
> I really hate to give up, but all I get from here is "it works" with nonsense references to non-related sample code...:/
>
>
>> Am 22.08.2021 um 17:54 schrieb Neil Young <foreverneilyoung at googlemail.com <mailto:foreverneilyoung at googlemail.com>>:
>>
>> @Nicolas Dufresne
>>
>> Would you mind to clarify in which wise this is an "answering app"?
>>
>> You said:
>>
>> <quote>
>> An answering app, I didn't link it before cause it's demoware. See the known issue about mdns candidates though, this is a missing feature that can pause problem.
>>
>> https://gitlab.collabora.com/showcases/ibc-2019/multistream-server <https://gitlab.collabora.com/showcases/ibc-2019/multistream-server>
>> </quote>
>>
>>
>> But in fact, it is not.
>>
>> Please clarify
>>
>>
>>
>>> Am 22.08.2021 um 16:11 schrieb Neil Young <foreverneilyoung at googlemail.com <mailto:foreverneilyoung at googlemail.com>>:
>>>
>>> > But nothing in the world makes webrtcbin answer an incoming H.264 offer only :))
>>>
>>> To be precise: To be able to work after having answered the offer. The answer comes, the video not.
>>>
>>>> Am 22.08.2021 um 16:05 schrieb Neil Young <foreverneilyoung at googlemail.com <mailto:foreverneilyoung at googlemail.com>>:
>>>>
>>>> BTW: I don't see the code you quoted doing handle an OFFER. Maybe I oversee something, but I don't see it emitting a "create-answer", which would be necessary to work.
>>>>
>>>> That being said I'm missing something like:
>>>>
>>>> self.webrtcbin.emit('create-answer', None, promise)
>>>>
>>>>
>>>> In C ofc
>>>>
>>>> But I might be stupid (admitted)
>>>>
>>>> Those samples doing "create-offer", "set-remote-description" bla bla... Well those work...
>>>>
>>>> But nothing in the world makes webrtcbin answer an incoming H.264 offer only :))
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Am 22.08.2021 um 15:33 schrieb Neil Young <foreverneilyoung at googlemail.com <mailto:foreverneilyoung at googlemail.com>>:
>>>>>
>>>>> Would it be possible to send me the content of a successful OFFER you are receiving from the outside to be processed by your app?
>>>>>
>>>>>
>>>>>> Am 22.08.2021 um 15:15 schrieb Neil Young <foreverneilyoung at googlemail.com <mailto:foreverneilyoung at googlemail.com>>:
>>>>>>
>>>>>> Could you show me an offer which is successfully handled by this app?
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am 22.08.2021 um 14:59 schrieb Neil Young <foreverneilyoung at googlemail.com <mailto:foreverneilyoung at googlemail.com>>:
>>>>>>>
>>>>>>> Thanks for sharing. Will check it out. I'm aware of the MDNS candidates thing, but
>>>>>>>
>>>>>>> a) I'm caring about those
>>>>>>> b) My Chrome is disabled to send them
>>>>>>>
>>>>>>> It is definitely not an ICE issue, I have
>>>>>>>
>>>>>>>> Am 22.08.2021 um 14:57 schrieb Nicolas Dufresne <nicolas at ndufresne.ca <mailto:nicolas at ndufresne.ca>>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le dim. 22 août 2021 08 h 48, Neil Young <foreverneilyoung at googlemail.com <mailto:foreverneilyoung at googlemail.com>> a écrit :
>>>>>>>> BTW: I will also be happy to hand out a POC app to everybody willing to help. You would only need to have a webrtcbin capable installation of GST and Python3. Works out of the box, no strings attached.
>>>>>>>>
>>>>>>>> An answering app, I didn't link it before cause it's demoware. See the known issue about mdns candidates though, this is a missing feature that can pause problem.
>>>>>>>>
>>>>>>>> https://gitlab.collabora.com/showcases/ibc-2019/multistream-server <https://gitlab.collabora.com/showcases/ibc-2019/multistream-server>
>>>>>>>>
>>>>>>>> In general, specially at this time of the year, a lot of folks are on vacation, leaving other really busy. Expecting someone to look into your issues for free in such a short timeframe is not realistic.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Am 22.08.2021 um 14:46 schrieb Neil Young <foreverneilyoung at googlemail.com <mailto:foreverneilyoung at googlemail.com>>:
>>>>>>>>>
>>>>>>>>> Well, I have published everything I did so far. I confirm, it works if webrtc/python is the offerer and it also works as described with an incoming offer, if the pipleline is using vp8 instead of H.264.
>>>>>>>>>
>>>>>>>>> But it does not no work with H.264. And unless anybody points me to the obvious mistake I have made with my handling there is nothing to be taken back with my conclusion.
>>>>>>>>>
>>>>>>>>> Prove me wrong, if you can. It doesn't help just to say "it works". There are very rare sample of answering apps with webrtcbin. And I'm doing nothing else then those.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Am 22.08.2021 um 14:41 schrieb Nicolas Dufresne <nicolas at ndufresne.ca <mailto:nicolas at ndufresne.ca>>:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Le dim. 22 août 2021 07 h 45, Neil Young via gstreamer-devel <gstreamer-devel at lists.freedesktop.org <mailto:gstreamer-devel at lists.freedesktop.org>> a écrit :
>>>>>>>>>> Finally giving up on this. Python, webrtcbin and H.264 offer - this does not work
>>>>>>>>>>
>>>>>>>>>> Hi Neil, I'm sorry this did not work you and no one was available in your required delay to freely support you.
>>>>>>>>>>
>>>>>>>>>> This answer is simply to state that this works for others, the webrtcbin and rtpbin pipeline are known to be difficult, but not broken (except for few missing webrtc features of course).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Am 21.08.2021 um 20:23 schrieb Neil Young <foreverneilyoung at googlemail.com <mailto:foreverneilyoung at googlemail.com>>:
>>>>>>>>>>>
>>>>>>>>>>> I'm hanging on this for a week now, and it seems so easy, but whatever I do, I can't make it work
>>>>>>>>>>>
>>>>>>>>>>> My app is a Python GST app, running on 18.4. It supports two modes: "call" and "answer". In call mode the app immediately issues an H.264 or VP8 OFFER (depending on a setting). This works in 100 % of the cases.
>>>>>>>>>>>
>>>>>>>>>>> What also works is when the app ANSWERs a VP8 offer. Just H.264 OFFERS are not working, even though the SDP answer looks good. I just don't get the video to remote.
>>>>>>>>>>>
>>>>>>>>>>> My app works like so:
>>>>>>>>>>>
>>>>>>>>>>> 1) When the app has a connection to the signaling server it creates a webrtcbin object:
>>>>>>>>>>>
>>>>>>>>>>> def build_webrtcbin(self):
>>>>>>>>>>> self.webrtcbin = Gst.ElementFactory.make("webrtcbin")
>>>>>>>>>>> self.webrtcbin.set_property("bundle-policy", "max-bundle")
>>>>>>>>>>> self.webrtcbin.set_property("stun-server", "stun://stun.l.google.com:19302 <>")
>>>>>>>>>>> self.webrtcbin.connect('on-negotiation-needed', self.on_negotiation_needed)
>>>>>>>>>>> self.webrtcbin.connect('on-ice-candidate', self.on_ice_candidate)
>>>>>>>>>>> self.webrtcbin.connect('pad-added', self.on_pad_added)
>>>>>>>>>>> self.webrtcbin.connect('notify::connection-state', self.on_connection_state)
>>>>>>>>>>> self.webrtcbin.connect('notify::signaling-state', self.on_signaling_state)
>>>>>>>>>>> self.webrtcbin.connect('notify::ice-connection-state', self.on_ice_connection_state)
>>>>>>>>>>> self.webrtcbin.connect('notify::ice-gathering-state', self.on_ice_gathering_state)
>>>>>>>>>>>
>>>>>>>>>>> self.pipeline = Gst.Pipeline.new()
>>>>>>>>>>> self.pipeline.add(self.webrtcbin)
>>>>>>>>>>>
>>>>>>>>>>> Nothing special I guess.
>>>>>>>>>>>
>>>>>>>>>>> 2) When the SDP OFFER comes in, the app does this:
>>>>>>>>>>>
>>>>>>>>>>> 2a) It starts the H.264 pipeline with `start_pipeline`.
>>>>>>>>>>> 2b) It emits the remote description to the local webrtcbin
>>>>>>>>>>>
>>>>>>>>>>> self.webrtcbin.emit('set-remote-description', desc, promise)
>>>>>>>>>>>
>>>>>>>>>>> 2c) It creates an answer calling `self.create_answer()`
>>>>>>>>>>>
>>>>>>>>>>> def create_answer(self):
>>>>>>>>>>> promise = Gst.Promise.new_with_change_func(self.on_answer_created, self.webrtcbin, None)
>>>>>>>>>>> self.webrtcbin.emit('create-answer', None, promise)
>>>>>>>>>>>
>>>>>>>>>>> def on_answer_created(self, promise, _, __):
>>>>>>>>>>> ''' Local WebRTC stack has created an answer '''
>>>>>>>>>>> promise.wait()
>>>>>>>>>>> reply = promise.get_reply()
>>>>>>>>>>> answer = reply.get_value('answer')
>>>>>>>>>>> promise = Gst.Promise.new()
>>>>>>>>>>> self.webrtcbin.emit('set-local-description', answer, promise)
>>>>>>>>>>> promise.interrupt()
>>>>>>>>>>> self.send_sdp_answer(answer) # Forwarding the answer to remote using the signaling server
>>>>>>>>>>>
>>>>>>>>>>> Observations so far:
>>>>>>>>>>>
>>>>>>>>>>> - It doesn't work if I don't set a payload-type which matches a payload type carried by the OFFERER or if I omit the payload-type. With ALL browsers it is sufficient to use the payload-type carried by the first H.264 element in the OFFER
>>>>>>>>>>> - It is not necessary to set the `profile-level-id`
>>>>>>>>>>>
>>>>>>>>>>> The best result I can achieve with this flow is a spinning wheel in the browser. No video.
>>>>>>>>>>>
>>>>>>>>>>> BUT:
>>>>>>>>>>>
>>>>>>>>>>> When I issue a call to `start_pipeline` somewhere shortly after the start (I use the moment, when I have a connection with the signaling server for this) with _exactly_the_same_payload_type_that_will_match_later_, I 100% have a video...
>>>>>>>>>>>
>>>>>>>>>>> This MUST be a side effect, because the log complains about the second `start_pipeline` (warnings, it seems to ignore it).... But it works. Of course this is not a working solution, because
>>>>>>>>>>>
>>>>>>>>>>> a) I would have to know the payload-type beforehand (it doesn't work if I use another payload-type in the first start)
>>>>>>>>>>> b) I would have to parse the incoming SDP for the first H.264 (which is doable)
>>>>>>>>>>>
>>>>>>>>>>> What am I doing wrong?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> def start_pipeline(self, payload_type):
>>>>>>>>>>> videotestsrc = Gst.ElementFactory.make("videotestsrc", "videotestsrc")
>>>>>>>>>>> videotestsrc.set_property("is-live", True)
>>>>>>>>>>> videotestsrc.set_property("pattern", "smpte")
>>>>>>>>>>>
>>>>>>>>>>> videotestsrc_caps = Gst.caps_from_string("video/x-raw, width=1280, height=720")
>>>>>>>>>>>
>>>>>>>>>>> videotestsrc_caps_filter = Gst.ElementFactory.make("capsfilter")
>>>>>>>>>>> videotestsrc_caps_filter.set_property("caps", videotestsrc_caps)
>>>>>>>>>>>
>>>>>>>>>>> videoconvert = Gst.ElementFactory.make("videoconvert", "videoconvert")
>>>>>>>>>>>
>>>>>>>>>>> x264enc = Gst.ElementFactory.make("x264enc", "x264enc")
>>>>>>>>>>> x264enc.set_property("threads", 4)
>>>>>>>>>>>
>>>>>>>>>>> x264enc_caps = Gst.caps_from_string("video/x-h264, profile=constrained-baseline")
>>>>>>>>>>>
>>>>>>>>>>> x264enc_caps_filter = Gst.ElementFactory.make("capsfilter")
>>>>>>>>>>> x264enc_caps_filter.set_property("caps", x264enc_caps)
>>>>>>>>>>>
>>>>>>>>>>> rtph264pay = Gst.ElementFactory.make("rtph264pay", "rtph264pay")
>>>>>>>>>>> rtph264pay.set_property("config-interval", -1)
>>>>>>>>>>> rtph264pay.set_property("aggregate-mode", "zero-latency")
>>>>>>>>>>>
>>>>>>>>>>> rtph264pay_caps = Gst.caps_from_string("application/x-rtp, media=video, encoding-name=H264")
>>>>>>>>>>> rtph264pay_caps.set_value("payload", payload_type)
>>>>>>>>>>>
>>>>>>>>>>> rtph264pay_caps_filter = Gst.ElementFactory.make("capsfilter")
>>>>>>>>>>> rtph264pay_caps_filter.set_property("caps", rtph264pay_caps)
>>>>>>>>>>>
>>>>>>>>>>> self.pipeline.add(videotestsrc)
>>>>>>>>>>> self.pipeline.add(videotestsrc_caps_filter)
>>>>>>>>>>> self.pipeline.add(videoconvert)
>>>>>>>>>>> self.pipeline.add(x264enc)
>>>>>>>>>>> self.pipeline.add(x264enc_caps_filter)
>>>>>>>>>>> self.pipeline.add(rtph264pay)
>>>>>>>>>>> self.pipeline.add(rtph264pay_caps_filter)
>>>>>>>>>>>
>>>>>>>>>>> Gst.Element.link <http://gst.element.link/>(videotestsrc, videotestsrc_caps_filter)
>>>>>>>>>>> Gst.Element.link <http://gst.element.link/>(videotestsrc_caps_filter, videoconvert)
>>>>>>>>>>> Gst.Element.link <http://gst.element.link/>(videoconvert, x264enc)
>>>>>>>>>>> Gst.Element.link <http://gst.element.link/>(x264enc, x264enc_caps_filter)
>>>>>>>>>>> Gst.Element.link <http://gst.element.link/>(x264enc_caps_filter, rtph264pay)
>>>>>>>>>>> Gst.Element.link <http://gst.element.link/>(rtph264pay, rtph264pay_caps_filter)
>>>>>>>>>>> Gst.Element.link <http://gst.element.link/>(rtph264pay_caps_filter, self.webrtcbin)
>>>>>>>>>>> self.pipeline.set_state(Gst.State.PLAYING)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210823/a34478ab/attachment-0001.htm>
More information about the gstreamer-devel
mailing list