Sync RTSP packets with broker messages

[tlpl]

Hi,

For the past few days, I am facing difficulties regarding RTSP streaming
along with sending additional information obtained from the stream. To
clarify picture following scenario:

Program A, running on machine 1, reads some sort of stream or a video file,
decodes it and perform some sort of analysis on incoming frames in the
middle of the pipeline. Analysis is done by a custom plugin implemented by
me, and results in obtaining some sort information, which can be nicely
packed into a string. Pipeline ends with "x264enc -> rtph264pay -> udpsink"
which is handled by GstRTSPServer in order to stream the contents.

Program B, running on machine 2, with pipeline "rtspsrc -> rtph264depay ->
avdec_h264 -> xvimagesink" reads the RTSP stream from Program A. RTSP
streaming from machine 1 to machine 2 works perfectly fine. Both programs
are done by me. No gst-launch is used.

The issue that I am facing is sending the result of the analysis done in
Program A to the Program B. Analysis is done for each individual frame (in
Program A), but it's consumed by corresponding frame in Program B.
Therefore, the result (string) must have some sort of unique identifier in
order to match it to the corresponding frame in Program B.

I have check "timestamp" value available on the rtph264pay element's src pad
in Program A and compared it to values on rtph264depay element's sink pad in
Program B. They match, and they potentially could serve as the unique
identifier to the message. However, since GstBuffer on rtph264 element is
different from the one during the analysis (obviously) I have no way of
knowing which message to pick (let's say I can store those messages in some
persistent storage). Sure, if they come in order (one by one) I can just
pick them as so, but that's not a robust solution, especially in case of
multiple input files / streams for Program A. Comparing GstBuffer on my
custom analysis plugin and the one on rtph264 sink / src pad also does not
show any seminaries that I could use (like pts / dts value).

Solutions I came up with (and their drawbacks): attaching analysis result to
the packets itself as some sort of metadata (same problem as above - I have
no way of knowing which analysis is corresponding to which buffer), setting
up some unique id by myself and encoding it on the actual frame before
sending and decoding it on the receiver (there must be a better way) and
finally "tagging" buffer during the analysis with unique id in order to add
this "tag" to packet's metadata and decode it on receiver (sure, but how). I
won't even mention that I have no idea how to implement any of the above if
those would be viable.

Not only I am completely stuck, but also, I have no idea how to formulate my
question concisely. The best I came up with is: how to sync incoming RTSP
packets with messages (strings) incoming from message broker for individual
video frames, while I have full control over both?

I really tried to make myself as clear as possible. Any notes and angry
comments are welcome.

Have a great day and thanks in advance!



--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/