Can't make webrtcbin connect to H.264 offering clients
Neil Young
foreverneilyoung at googlemail.com
Sat Aug 21 18:23:01 UTC 2021
I'm hanging on this for a week now, and it seems so easy, but whatever I do, I can't make it work
My app is a Python GST app, running on 18.4. It supports two modes: "call" and "answer". In call mode the app immediately issues an H.264 or VP8 OFFER (depending on a setting). This works in 100 % of the cases.
What also works is when the app ANSWERs a VP8 offer. Just H.264 OFFERS are not working, even though the SDP answer looks good. I just don't get the video to remote.
My app works like so:
1) When the app has a connection to the signaling server it creates a webrtcbin object:
def build_webrtcbin(self):
self.webrtcbin = Gst.ElementFactory.make("webrtcbin")
self.webrtcbin.set_property("bundle-policy", "max-bundle")
self.webrtcbin.set_property("stun-server", "stun://stun.l.google.com:19302")
self.webrtcbin.connect('on-negotiation-needed', self.on_negotiation_needed)
self.webrtcbin.connect('on-ice-candidate', self.on_ice_candidate)
self.webrtcbin.connect('pad-added', self.on_pad_added)
self.webrtcbin.connect('notify::connection-state', self.on_connection_state)
self.webrtcbin.connect('notify::signaling-state', self.on_signaling_state)
self.webrtcbin.connect('notify::ice-connection-state', self.on_ice_connection_state)
self.webrtcbin.connect('notify::ice-gathering-state', self.on_ice_gathering_state)
self.pipeline = Gst.Pipeline.new()
self.pipeline.add(self.webrtcbin)
Nothing special I guess.
2) When the SDP OFFER comes in, the app does this:
2a) It starts the H.264 pipeline with `start_pipeline`.
2b) It emits the remote description to the local webrtcbin
self.webrtcbin.emit('set-remote-description', desc, promise)
2c) It creates an answer calling `self.create_answer()`
def create_answer(self):
promise = Gst.Promise.new_with_change_func(self.on_answer_created, self.webrtcbin, None)
self.webrtcbin.emit('create-answer', None, promise)
def on_answer_created(self, promise, _, __):
''' Local WebRTC stack has created an answer '''
promise.wait()
reply = promise.get_reply()
answer = reply.get_value('answer')
promise = Gst.Promise.new()
self.webrtcbin.emit('set-local-description', answer, promise)
promise.interrupt()
self.send_sdp_answer(answer) # Forwarding the answer to remote using the signaling server
Observations so far:
- It doesn't work if I don't set a payload-type which matches a payload type carried by the OFFERER or if I omit the payload-type. With ALL browsers it is sufficient to use the payload-type carried by the first H.264 element in the OFFER
- It is not necessary to set the `profile-level-id`
The best result I can achieve with this flow is a spinning wheel in the browser. No video.
BUT:
When I issue a call to `start_pipeline` somewhere shortly after the start (I use the moment, when I have a connection with the signaling server for this) with _exactly_the_same_payload_type_that_will_match_later_, I 100% have a video...
This MUST be a side effect, because the log complains about the second `start_pipeline` (warnings, it seems to ignore it).... But it works. Of course this is not a working solution, because
a) I would have to know the payload-type beforehand (it doesn't work if I use another payload-type in the first start)
b) I would have to parse the incoming SDP for the first H.264 (which is doable)
What am I doing wrong?
def start_pipeline(self, payload_type):
videotestsrc = Gst.ElementFactory.make("videotestsrc", "videotestsrc")
videotestsrc.set_property("is-live", True)
videotestsrc.set_property("pattern", "smpte")
videotestsrc_caps = Gst.caps_from_string("video/x-raw, width=1280, height=720")
videotestsrc_caps_filter = Gst.ElementFactory.make("capsfilter")
videotestsrc_caps_filter.set_property("caps", videotestsrc_caps)
videoconvert = Gst.ElementFactory.make("videoconvert", "videoconvert")
x264enc = Gst.ElementFactory.make("x264enc", "x264enc")
x264enc.set_property("threads", 4)
x264enc_caps = Gst.caps_from_string("video/x-h264, profile=constrained-baseline")
x264enc_caps_filter = Gst.ElementFactory.make("capsfilter")
x264enc_caps_filter.set_property("caps", x264enc_caps)
rtph264pay = Gst.ElementFactory.make("rtph264pay", "rtph264pay")
rtph264pay.set_property("config-interval", -1)
rtph264pay.set_property("aggregate-mode", "zero-latency")
rtph264pay_caps = Gst.caps_from_string("application/x-rtp, media=video, encoding-name=H264")
rtph264pay_caps.set_value("payload", payload_type)
rtph264pay_caps_filter = Gst.ElementFactory.make("capsfilter")
rtph264pay_caps_filter.set_property("caps", rtph264pay_caps)
self.pipeline.add(videotestsrc)
self.pipeline.add(videotestsrc_caps_filter)
self.pipeline.add(videoconvert)
self.pipeline.add(x264enc)
self.pipeline.add(x264enc_caps_filter)
self.pipeline.add(rtph264pay)
self.pipeline.add(rtph264pay_caps_filter)
Gst.Element.link(videotestsrc, videotestsrc_caps_filter)
Gst.Element.link(videotestsrc_caps_filter, videoconvert)
Gst.Element.link(videoconvert, x264enc)
Gst.Element.link(x264enc, x264enc_caps_filter)
Gst.Element.link(x264enc_caps_filter, rtph264pay)
Gst.Element.link(rtph264pay, rtph264pay_caps_filter)
Gst.Element.link(rtph264pay_caps_filter, self.webrtcbin)
self.pipeline.set_state(Gst.State.PLAYING)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210821/3ded4653/attachment-0001.htm>
More information about the gstreamer-devel
mailing list