Re: Efficient pipeline for streaming BGR frames to the local network via UDP

[Michael Gruner]

Hi

Most of the latency comes from the encoding/decoding pair and the Transport Stream standard.

1. Set the following properties to the x264enc:

x264enc speed-preset=ultrafast tune=zerolatency

2. The TS receiver will typically add 700ms of latency (as specified by the standard). If you have control over the receiver, try to see if you can remove that. In GStreamer you would do:

tsdemux latency=0 # or whatever number you find appropriate for your application. In video, 40ms is typically enough

3. I don't think you rndbuffersize anymore. I remember that being a hackish way to group TS packets and be sent more efficiently leveraging the UDP MTU. Later on, the alignment property in the mpegtsmux was added to fulfill that purpose.

4. OpenCV will perform a memory copy when receiving a matrix from the VideoCapture and when sending a matrix to the VideoWriter. You may be able to save a few milliseconds if you use GStreamer directly (as opposed to using the OpenCV wrapper).

5. Again, if you have control over the client, make sure you check on the caching settings. Most clients do some sort of buffering to enhance viewing experience.

Finally, please be aware that support now is typically handled in the discourse server:
https://discourse.gstreamer.org/
GStreamer Discourse
discourse.gstreamer.org


Hope this helps!
Michael
www.ridgerun.com

> On 21 Aug 2024, at 11:38 AM, Ravi Joshi via gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
> 
> Hello,
> 
> First of all, thank you very much for allowing me to post here. I am receiving h264 frames of a stream (720x360 at 30fps) from a camera connected to my PC via USB. I am converting these frames to BGR frames supported in OpenCV. After the conversion, these frames are sent to the local network via UDP using GStreamer in C++. A code snippet is shown below:
> 
> 
> ---------------------------------------
> class VideoServer {
> public:
>   VideoServer() {
>     std::string pipeline = "appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! x264enc bitrate=5000 ! "
>                            "mpegtsmux alignment=7 ! rndbuffersize max=1316 min=1316 ! udpsink port=5005";
> 
>     writer = std::make_unique<cv::VideoWriter>(pipeline, cv::CAP_GSTREAMER, 0, fps_, cv::Size(width_, height_));
>   }
>   void VideoDataCallack(const uint8_t *data, size_t size, int64_t timestamp) override {
>     std::cout << "At timestamp: " << timestamp << " video data of size " << size << " received." << std::endl;
>     // convert h264_frame to opencv_frame
>     writer->write(opencv_frame);
>   }
> 
> private:
>   std::unique_ptr<cv::VideoWriter> writer;
> };
> ---------------------------------------
> 
> 
> My goal is to steam these frames with minimal latency. Therefore, can someone point me to an efficient pipeline suitable for the above code snippet?
> 
> Thank you for your time and kind support.
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20240821/3103eee4/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: b39794bde7833cc5510435c45db50a001b109713.png
Type: image/png
Size: 22554 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20240821/3103eee4/attachment-0001.png>