WebRTC with frame-synchronized metadata

[Nicolas Dufresne]

Le mer. 2 janv. 2019 18 h 11, Jesse Gray <jg at ifrobots.com> a écrit :

> Hello,
>
> We are interested in sending frame-synchronized data along with the video
> frames in a WebRTC stream (using GStreamer's webrtcbin).  For example,
> consider a moving camera: it would be useful to include the current
> coordinates of the camera along with the video frame.  Or in a computer
> vision scenario, we might want to send the rectangle bounding some detected
> object in that specific frame.  In each case we want the receiver of a
> WebRTC stream to be able to match up the metadata to the same video frame
> as intended by the sender.
>
> The most convenient solution would be if the sender could somehow attach
> the metadata to the frame so it arrives at the receiver associated with the
> correct frame.  But, also workable is just having a mechanism to identify
> the frames uniquely across sender/receiver, and then transmit the metadata
> some other way keyed off that unique identity.
>
> Does anyone know if there is a way to attach metadata to an image frame in
> such a way that it is accessible to the receiver after traveling over the
> WebRTC connection?
>
> Or, is there any unique ID associated with an image frame on the sending
> side that is preserved over the WebRTC link to the receiver that could be
> used as a key to associate data with that specific frame?  For example, is
> there a timestamp value that is the same on both ends?
>
> It would be great to find a solution that works between GStreamer and a
> browser, but a solution that relies on GStreamer's webrtcbin managing both
> ends of the connection is also very interesting!
>

I'm not sure for the browser part, but GStreamer to GStreamer, we have
successfully used RTP header extension. See GstRTPBuffer API, there is
helpers to add such extension to RTP packets. We were using pad probes to
attach the information.  One to detect new frame, and another to attach the
information on all packets of this frame. All in case a packet is lost.


> Thanks very much for any ideas or suggestions!
> Jesse
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20190102/f19079d0/attachment.html>