NV12 -> GRAY8 conversion (BT601 vs BT709)
Nicolas Dufresne
nicolas at ndufresne.ca
Wed Feb 3 15:36:06 UTC 2021
Le mercredi 03 février 2021 à 06:08 -0600, Marianna S. Buschle a écrit :
> I'm getting some behavior I don't understand when converting from NV12 to
> GRAY8 (as a way of extracting the intensity channel).
>
> I'm using gstreamer to compare the "correctness" of the RGB->NV12 conversion
> that the driver from our proprietary camera is doing.
>
> What is puzzling me is the differences I see in how BT601 and BT709 are
> being threated.
This is indeed strange, I would expdect similar behaviour. Can you check what
colorimetry get negotiated for the GRAY8 output ? Videoconvert should support
both limited and full range GRAY8 and this "should" be reflected in the
colorimetry field. Note that field name like "bt709" are short cust, the caps
fields can also be selected with numbers for each of the item in
GstVideoColorimetry (iirc syntax is <range>:<matrix>:<transfer>:<primaries>).
>
> If I run:
> ```
> GST_DEBUG="*:3,*video-color*:5,*video-converter*:5" gst-launch-1.0 v4l2src
> io-mode=2 hw-center-crop=true device=/dev/qt5023_video0 num-buffers=1 !
> "video/x-raw,width=1920,height=1080,format=RGB,framerate=30/1" !
> videoconvert ! "video/x-raw,format=NV12,colorimetry=bt709" ! videoconvert !
> "video/x-raw,format=GRAY8" ! fakesink 2>&1 | grep color_matrix
> ```
>
> ```
> 0:00:00.028992885 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.003922 0.000000 0.000000
> 0.000000]
> 0:00:00.029011478 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [0.000000 0.003922 0.000000
> 0.000000]
> 0:00:00.029027743 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.000000 0.000000 0.003922
> 0.000000]
> 0:00:00.029046690 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.029071710 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.003922 0.000000 0.000000
> 0.000000]
> 0:00:00.029088413 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [0.000000 0.003922 0.000000
> 0.000000]
> 0:00:00.029104050 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.000000 0.000000 0.003922
> 0.000000]
> 0:00:00.029120480 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.029146813 73121 0x5635afe43a30 DEBUG video-color
> video-color.c:392:gst_video_color_matrix_get_Kr_Kb: matrix: 3, Kr 0.212600,
> Kb 0.072200
> 0:00:00.029198289 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.182586 0.614231 0.062007
> 16.000000]
> 0:00:00.029223928 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [-0.100644 -0.338572 0.439216
> 128.000000]
> 0:00:00.029241376 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.439216 -0.398942 -0.040274
> 128.000000]
> 0:00:00.029257076 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.029282192 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.182586 0.614231 0.062007
> 16.000000]
> 0:00:00.029299343 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [-0.100644 -0.338572 0.439216
> 128.000000]
> 0:00:00.029327055 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.439216 -0.398942 -0.040274
> 128.000000]
> 0:00:00.029344473 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.029362300 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1026:color_matrix_convert: [ 47 157 16 4096]
> 0:00:00.029377440 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1028:color_matrix_convert: [ -26 -87 112 32768]
> 0:00:00.029392319 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1030:color_matrix_convert: [ 112 -102 -10 32768]
> 0:00:00.029406848 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1032:color_matrix_convert: [ 0 0 0 1]
> 0:00:00.030482502 73121 0x5635afe43a30 DEBUG video-color
> video-color.c:392:gst_video_color_matrix_get_Kr_Kb: matrix: 3, Kr 0.212600,
> Kb 0.072200
> 0:00:00.030501748 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.004566 0.000000 0.007030
> -0.972945]
> 0:00:00.030516922 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [0.004566 -0.000836 -0.002090
> 0.301483]
> 0:00:00.030533715 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.004566 0.008284 0.000000
> -1.133402]
> 0:00:00.030549550 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.030574723 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.004566 0.000000 0.007030
> -0.972945]
> 0:00:00.030593849 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [0.004566 -0.000836 -0.002090
> 0.301483]
> 0:00:00.030609228 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.004566 0.008284 0.000000
> -1.133402]
> 0:00:00.030626048 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.030650578 73121 0x5635afe43a30 DEBUG video-color
> video-color.c:392:gst_video_color_matrix_get_Kr_Kb: matrix: 4, Kr 0.299000,
> Kb 0.114000
> 0:00:00.030700063 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [1.164384 0.115637 0.223212
> -62.002768]
> 0:00:00.030717142 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [-0.000000 1.126843 -0.125966
> -0.112195]
> 0:00:00.030731795 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.000000 -0.082480 1.119493
> -4.737686]
> 0:00:00.030750646 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.030774863 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [1.164384 0.115637 0.223212
> -62.002768]
> 0:00:00.030793160 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [-0.000000 1.126843 -0.125966
> -0.112195]
> 0:00:00.030809265 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.000000 -0.082480 1.119493
> -4.737686]
> 0:00:00.030823683 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.030840235 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1026:color_matrix_convert: [ 298 30 57 -15873]
> 0:00:00.030856657 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1028:color_matrix_convert: [ 0 288 -32 -29]
> 0:00:00.030869308 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1030:color_matrix_convert: [ 0 -21 287 -1213]
> 0:00:00.030882449 73121 0x5635afe43a30 DEBUG video-converter
> video-converter.c:1032:color_matrix_convert: [ 0 0 0 1]
> ```
>
> I can see in the log that several conversions are made, from RGB->NV12 and
> what I guess is from NV12->RGB->GRAY8?
> And if I save the resulting frame to fx a PNM I get something that looks
> like the expected, I have full range [0:255] the frame is quite close to
> what I get from converting from the input RGB->GRAY8 and if I look at the
> histogram I can see that it seems to have been "stretched" to get the full
> range from the limited range [16:235].
>
> Now if I do the same but specify the NV12 colorimetry to bt601 instead:
> ```
> root at qt5122:~# GST_DEBUG="*:3,*video-color*:5,*video-converter*:5"
> gst-launch-1.0 v4l2src io-mode=2 hw-center-crop=true
> device=/dev/qt5023_video0 num-buffers=1 !
> "video/x-raw,width=1920,height=1080,format=RGB,framerate=30/1" !
> videoconvert ! "video/x-raw,format=NV12,colorimetry=bt601" ! videoconvert !
> "video/x-raw,format=GRAY8" ! fakesink 2>&1 | grep color_matrix
> ```
>
> ```
> 0:00:00.028712681 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.003922 0.000000 0.000000
> 0.000000]
> 0:00:00.028731995 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [0.000000 0.003922 0.000000
> 0.000000]
> 0:00:00.028748744 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.000000 0.000000 0.003922
> 0.000000]
> 0:00:00.028763320 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.028795701 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.003922 0.000000 0.000000
> 0.000000]
> 0:00:00.028811961 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [0.000000 0.003922 0.000000
> 0.000000]
> 0:00:00.028828029 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.000000 0.000000 0.003922
> 0.000000]
> 0:00:00.028842601 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.028868628 73153 0x5650e5f6ea30 DEBUG video-color
> video-color.c:392:gst_video_color_matrix_get_Kr_Kb: matrix: 4, Kr 0.299000,
> Kb 0.114000
> 0:00:00.028917500 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.256788 0.504129 0.097906
> 16.000000]
> 0:00:00.028935040 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [-0.148223 -0.290993 0.439216
> 128.000000]
> 0:00:00.028951602 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.439216 -0.367788 -0.071427
> 128.000000]
> 0:00:00.028967314 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.028997272 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1007:color_matrix_debug: [0.256788 0.504129 0.097906
> 16.000000]
> 0:00:00.029016155 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1009:color_matrix_debug: [-0.148223 -0.290993 0.439216
> 128.000000]
> 0:00:00.029032904 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1011:color_matrix_debug: [0.439216 -0.367788 -0.071427
> 128.000000]
> 0:00:00.029048794 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
> 1.000000]
> 0:00:00.029066010 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1026:color_matrix_convert: [ 66 129 25 4096]
> 0:00:00.029080734 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1028:color_matrix_convert: [ -38 -74 112 32768]
> 0:00:00.029094831 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1030:color_matrix_convert: [ 112 -94 -18 32768]
> 0:00:00.029109267 73153 0x5650e5f6ea30 DEBUG video-converter
> video-converter.c:1032:color_matrix_convert: [ 0 0 0 1]
> ```
>
> Here I only seem to be able to see the NV12->GRAY8 conversion.
> And I'm guessing the GRAY is just the pure Y extracted from the NV12.
> But that results in a gray image where the blacks are grayish and if I look
> at the histogram I can see I only have the limited range [16:235], which
> explains the grayness.
>
> I expected that here we also need to contrast stretch the gray image in
> order to get back the full range [0:255].
>
> So why the difference in how gstreamer threats the gray conversion depending
> on the matrix conversion?
> Is this a bug or is it because I misunderstood something in these standards?
>
>
>
> --
> Sent from: http://gstreamer-devel.966125.n4.nabble.com/
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
More information about the gstreamer-devel
mailing list