[EXTERN] HW de/encoding & Chroma supsampling

[Michael Scherle]

On 29.07.24 13:56, Michael Scherle wrote:
> Hello,
> 
> with the hopefully soon merged pull request: “[PATCH v8 0/6] dcc: Create 
> a stream for non-gl/remote clients that want to use dmabuf (v8)” (which 
> we urgently need for our project), I have noticed the problem of chroma 
> supsampling. The problem usually occurs with colored text and thin 
> lines. Which of course is bad in a remote desktop scenario.
> 
> The attached screenshot shows a test image of rtings on chroma 
> supsampling, whereby it should be noted that the image should either be 
> viewed at exactly 100% or greatly enlarged, as otherwise artifacts may 
> occur that are not present in the original.
> 
> The left third shows the current hardware encoding implementation, which 
> uses Chroma Supsampling 4:2:0, and the resulting artifacts that occur 
> especially in blue and red. In the right third you can see a prototype 
> implementation of 4:4:4 hardware de/encoding that produces a perfect 
> image. In the center is a prototype of 4:2:0 encoding, where the image 
> is upsampled to double before encoding and downsampled after decoding to 
> remove the artifacts. The picture does not show the artifacts, but it is 
> a bit blurry due to the interpolation of the up/downsampling of the 
> Gstreamer plugins. I have modified the Gstreamer plugins so that it is 
> possible to set the interpolation method to Nearest Neighbor, then the 
> blurring is gone and the picture is the same as 4:4:4 (but of course it 
> has the disadvantage of higher bandwidths and en/decoder load). 
> Nevertheless, in some cases it could be a fallback for hardware that 
> cannot work with 4:4:4.
> You can find the original test image here: 
> https://www.rtings.com/images/test-materials/2017/chroma-444.png if you 
> want to try it out for yourself.
> 
> 
> Since I would like to submit my implementation after the above merge 
> request has been merged (since it depends on it), I have a few questions 
> for a reasonable implementation.
> 
> At the moment I have implemented everything as a separate format, e.g. 
> for h265:
> 
>   SPICE_VIDEO_CODEC_TYPE_H265,
>   SPICE_VIDEO_CODEC_TYPE_H265_444,
>   SPICE_VIDEO_CODEC_TYPE_H265_U,
> 
> and for caps:
> 
>   SPICE_DISPLAY_CAP_CODEC_H265,
>   SPICE_DISPLAY_CAP_CODEC_H265_444,
>   SPICE_DISPLAY_CAP_CODEC_H265_U,
> 
> For caps, I think this makes sense, since a HW de/encoder, for example, 
> could only do 4:2:0. But what do you think, does it also make sense for 
> the formats or should it be done via extra parameters?
> 
> Where should I place my fork? should it be on the freedesktop gitlab? 
> then I would have to see how to get the appropriate permissions.
> 
> Does anyone know of any other methods of avoiding artifacts on hardware 
> that can only work with chroma supsamling?
> 
> 
> Greetings
> Michael


Hello,

I got the patches for gstreamer merged for the va and msdk plugin for 
Nearest Neighbor. With them the upsampling looks flawlessly.

Regarding merge requests, are these still accepted at all, for example:
 >dcc: Create a stream for non-gl/remote clients that want to use dmabuf 
(v8)
doesn't seem to be handled anymore?
can you still make merge requestsor are only security vulnerabilities fixed?

Greetings
Michael