How to send a series of zipped bitmaps files / raw video frames by tcp with gstreamer?

Tue Jun 16 17:43:04 UTC 2020

Le mardi 16 juin 2020 à 11:07 +0800, F32 a écrit :
> Here's more detailed background information:
> 
> I have a 3D (OpenGL) rendering software running on server A, and a
> media server based on gstreamer running on server B.
> 
> Server A is also the start point of the gstreamer pipeline. It
> fetches raw images from OpenGL, and puts images into gstreamer (e.g.,
> the tcpclientsink element)
> 
> H264 / H265 is not used because it consumes a lot of cpu / gpu
> memory. However, sending raw images over network also consumes a lot
> of cpu / bandwidth, because raw images are just too large.
> 
> A trade-off is to use a lightweight compression algorithm (e.g., lz4
> -1) which can compress 1GB bitmaps / raw images per second, per core.
> In my case, file size can be reduced by 90%.
> 
> Too achieve this, I'm thinking about:
>  - creating a lz4enc element, which puts "zipped content size" +
> "zipped content" into a gstbuffer, and then sends it by tcp
>  - creating a lz4dec element, which keeps reading bytes from
> tcpserversrc, and outputs a decompressed gstbuffer when both "zipped
> content size" and "zipped content" are received
> Is it the correct way to achieve this?

Perhaps you'd like to seek inspiration in PNG (or just use that). It's
a simple libz based RGB tile compressor, but to support the P
(portable) they introduce a magic cookie and pixel layout / image size
information. With this design, the receiver can:

  - Figure-out the image size and format (signalling free)
  - Decompress as data flows (tiles) or on multiple cores
  - Use low CPU (low complexity)
  - It's truly loss-less

Its bad for lossy network and do not have RTP integration (though
GStreamer offer a GST specific generaic payloader). It also does not
have lossy mechanism to be able to guaranty a compression budget. Of
course, there is other solution to like old JPEG 2K, Apple ProRES, RED
encoding, NDI (warning, bad reputation ;-P), etc. They are all about
visually loss-less/low-complexity and fixed budget networks/storage
speed.

> 
> (Also, I think that lightweight compression / decompression like lz4
> is an interesting feature that might be officially added to gstreamer
> in the long term)

Adding niche and unique compression format, rather then using widely
adopted standard is not that interesting for maintainers. But if you
host your own project with a library and documentation, and provide a
plugin using that, this would be acceptable for upstream GStreamer
inclusion. Hosting your own plugins as part of your project is of
course totally fine ;-P

Instead of your own project, you could add to FFMPEG CODEC collection,
which unlike GStreamer is meant to offer compression algorithm directly
(clean room implementation, patent encumbered is apparently fine
there).

> 
> At 2020-06-15 21:46:40, "Nicolas Dufresne" <nicolas at ndufresne.ca>
> wrote:
> >Le lundi 15 juin 2020 à 17:28 +0800, F32 a écrit :
> >> Hi,
> >> 
> >> I'm trying to send a series of zipped bitmap files / raw frames
> over
> >> network (because it consumes much less cpu than h264).
> >> The following command works and all files are copied correctly:
> >> 
> >> > gst-launch-1.0 multifilesrc location="/dev/shm/1/%05d.bmp"
> index=1
> >> ! multifilesink location="%05d.bmp"
> >> 
> >> However, when I tried to transfer these files over tcp:
> >> 
> >>  > gst-launch-1.0 multifilesrc location="%05d.bmp" index=1 !
> >> tcpclientsink port=3000
> >>  > gst-launch-1.0 tcpserversrc port=3000 ! multifilesink
> >> location="%05d.bmp"
> >> 
> >> The result is incorrect. What I got is numerous 4KB files instead
> of
> >> large bitmap files.
> >> 
> >> I have learned that for raw video over tcp, we need to add a
> >> "rawvideoparse" element after tcpserversrc, but it does not seem
> to
> >> support generic compression (zip, gzip, etc.)
> >> 
> >> What is the simplest way to support transferring a series of
> zipped
> >> bitmap files / raw video frames by tcp with gstreamer? Do I need
> to
> >> extend the rawvideoaprse, write a special zipdec element, or
> >> something else?
> >
> >The answer will depend on what you really intend to do. If generic
> file
> >transfer is your end goal, this is simply the wrong software ;-P But
> if
> >you do have some streaming use cases around that, we can likely
> help,
> >assuming we have more details. Of course, when you chose a streaming
> >method that is not following any existing standard, you might endup
> >having to implement some elements for it.
> >
> >> 
> >> Regards,
> >> Windy
> >> 
> >> 
> >>  
> >> 
> >> _______________________________________________
> >> gstreamer-devel mailing list
> >> gstreamer-devel at lists.freedesktop.org
> >> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> >
> >_______________________________________________
> >gstreamer-devel mailing list
> >gstreamer-devel at lists.freedesktop.org
> >https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> 
> 
>  
> 
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel