raw video stream with per frame meta data over network

[Thomas Johansson]

Hello all, I'm Thomas Johansson and working on a problem involving a raw
video stream with associated meta data.
I need to transfer a stream of raw images from a video4linux2 source with a
set of per frame data fields over UDP, The meta data will be different for
each and every frame.

Refer to the RFC3550 for RTP transport, the RFC4175 for transferring raw
images (YCbCr 4:2:2) and RFC6597 (KLV data) for the associated meta data
and as of the spec for RFC4175:
"This payload format requires that applications using such synchronized
ancillary
   data SHOULD deliver it in separate RTP sessions that operate
   concurrently with the video session.  The normal RTP mechanisms
   SHOULD be used to synchronize the media."

I have made an attempt to build a gstreamer rtpbin solution but I'm unsure
of the proper way to do it.

Question 1, in the world of gstreamer, what would be the natural way to
transfer meta data and associate it with each frame ? In my solution I'm
using the buffer-mode" setting on the receivers jitterbuffer to force the
PTS timestamp to follow the buffers PTS on the sending side.

Question 2, I'm using the KLV rtp payloader and the appsource element,
however I'm unsure of how to actually send a new packet of KLV meta data
for each frame. My first attempt was to use a Identity element of the
video4linux source and at every handoff transfer a packet of the KLV,
however it fails and I don't know how to properly set up the transfer.

I would appreciate if someone could assist me with a proper design of the
overall problem, the issue of attaching frame specific data with each and
every frame over a UDP link. Also, if the KLV is the way to go, I'm having
trouble using the appsource for KLV so any pointers would be good.

Thanks
/Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20181218/79d5e2dc/attachment.html>