NV12 -> GRAY8 conversion (BT601 vs BT709)
Marianna S. Buschle
msb at qtec.com
Wed Feb 3 12:08:50 UTC 2021
I'm getting some behavior I don't understand when converting from NV12 to
GRAY8 (as a way of extracting the intensity channel).
I'm using gstreamer to compare the "correctness" of the RGB->NV12 conversion
that the driver from our proprietary camera is doing.
What is puzzling me is the differences I see in how BT601 and BT709 are
being threated.
If I run:
```
GST_DEBUG="*:3,*video-color*:5,*video-converter*:5" gst-launch-1.0 v4l2src
io-mode=2 hw-center-crop=true device=/dev/qt5023_video0 num-buffers=1 !
"video/x-raw,width=1920,height=1080,format=RGB,framerate=30/1" !
videoconvert ! "video/x-raw,format=NV12,colorimetry=bt709" ! videoconvert !
"video/x-raw,format=GRAY8" ! fakesink 2>&1 | grep color_matrix
```
```
0:00:00.028992885 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.003922 0.000000 0.000000
0.000000]
0:00:00.029011478 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [0.000000 0.003922 0.000000
0.000000]
0:00:00.029027743 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.000000 0.000000 0.003922
0.000000]
0:00:00.029046690 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.029071710 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.003922 0.000000 0.000000
0.000000]
0:00:00.029088413 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [0.000000 0.003922 0.000000
0.000000]
0:00:00.029104050 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.000000 0.000000 0.003922
0.000000]
0:00:00.029120480 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.029146813 73121 0x5635afe43a30 DEBUG video-color
video-color.c:392:gst_video_color_matrix_get_Kr_Kb: matrix: 3, Kr 0.212600,
Kb 0.072200
0:00:00.029198289 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.182586 0.614231 0.062007
16.000000]
0:00:00.029223928 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [-0.100644 -0.338572 0.439216
128.000000]
0:00:00.029241376 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.439216 -0.398942 -0.040274
128.000000]
0:00:00.029257076 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.029282192 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.182586 0.614231 0.062007
16.000000]
0:00:00.029299343 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [-0.100644 -0.338572 0.439216
128.000000]
0:00:00.029327055 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.439216 -0.398942 -0.040274
128.000000]
0:00:00.029344473 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.029362300 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1026:color_matrix_convert: [ 47 157 16 4096]
0:00:00.029377440 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1028:color_matrix_convert: [ -26 -87 112 32768]
0:00:00.029392319 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1030:color_matrix_convert: [ 112 -102 -10 32768]
0:00:00.029406848 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1032:color_matrix_convert: [ 0 0 0 1]
0:00:00.030482502 73121 0x5635afe43a30 DEBUG video-color
video-color.c:392:gst_video_color_matrix_get_Kr_Kb: matrix: 3, Kr 0.212600,
Kb 0.072200
0:00:00.030501748 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.004566 0.000000 0.007030
-0.972945]
0:00:00.030516922 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [0.004566 -0.000836 -0.002090
0.301483]
0:00:00.030533715 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.004566 0.008284 0.000000
-1.133402]
0:00:00.030549550 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.030574723 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.004566 0.000000 0.007030
-0.972945]
0:00:00.030593849 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [0.004566 -0.000836 -0.002090
0.301483]
0:00:00.030609228 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.004566 0.008284 0.000000
-1.133402]
0:00:00.030626048 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.030650578 73121 0x5635afe43a30 DEBUG video-color
video-color.c:392:gst_video_color_matrix_get_Kr_Kb: matrix: 4, Kr 0.299000,
Kb 0.114000
0:00:00.030700063 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [1.164384 0.115637 0.223212
-62.002768]
0:00:00.030717142 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [-0.000000 1.126843 -0.125966
-0.112195]
0:00:00.030731795 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.000000 -0.082480 1.119493
-4.737686]
0:00:00.030750646 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.030774863 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [1.164384 0.115637 0.223212
-62.002768]
0:00:00.030793160 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [-0.000000 1.126843 -0.125966
-0.112195]
0:00:00.030809265 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.000000 -0.082480 1.119493
-4.737686]
0:00:00.030823683 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.030840235 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1026:color_matrix_convert: [ 298 30 57 -15873]
0:00:00.030856657 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1028:color_matrix_convert: [ 0 288 -32 -29]
0:00:00.030869308 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1030:color_matrix_convert: [ 0 -21 287 -1213]
0:00:00.030882449 73121 0x5635afe43a30 DEBUG video-converter
video-converter.c:1032:color_matrix_convert: [ 0 0 0 1]
```
I can see in the log that several conversions are made, from RGB->NV12 and
what I guess is from NV12->RGB->GRAY8?
And if I save the resulting frame to fx a PNM I get something that looks
like the expected, I have full range [0:255] the frame is quite close to
what I get from converting from the input RGB->GRAY8 and if I look at the
histogram I can see that it seems to have been "stretched" to get the full
range from the limited range [16:235].
Now if I do the same but specify the NV12 colorimetry to bt601 instead:
```
root at qt5122:~# GST_DEBUG="*:3,*video-color*:5,*video-converter*:5"
gst-launch-1.0 v4l2src io-mode=2 hw-center-crop=true
device=/dev/qt5023_video0 num-buffers=1 !
"video/x-raw,width=1920,height=1080,format=RGB,framerate=30/1" !
videoconvert ! "video/x-raw,format=NV12,colorimetry=bt601" ! videoconvert !
"video/x-raw,format=GRAY8" ! fakesink 2>&1 | grep color_matrix
```
```
0:00:00.028712681 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.003922 0.000000 0.000000
0.000000]
0:00:00.028731995 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [0.000000 0.003922 0.000000
0.000000]
0:00:00.028748744 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.000000 0.000000 0.003922
0.000000]
0:00:00.028763320 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.028795701 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.003922 0.000000 0.000000
0.000000]
0:00:00.028811961 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [0.000000 0.003922 0.000000
0.000000]
0:00:00.028828029 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.000000 0.000000 0.003922
0.000000]
0:00:00.028842601 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.028868628 73153 0x5650e5f6ea30 DEBUG video-color
video-color.c:392:gst_video_color_matrix_get_Kr_Kb: matrix: 4, Kr 0.299000,
Kb 0.114000
0:00:00.028917500 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.256788 0.504129 0.097906
16.000000]
0:00:00.028935040 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [-0.148223 -0.290993 0.439216
128.000000]
0:00:00.028951602 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.439216 -0.367788 -0.071427
128.000000]
0:00:00.028967314 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.028997272 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1007:color_matrix_debug: [0.256788 0.504129 0.097906
16.000000]
0:00:00.029016155 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1009:color_matrix_debug: [-0.148223 -0.290993 0.439216
128.000000]
0:00:00.029032904 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1011:color_matrix_debug: [0.439216 -0.367788 -0.071427
128.000000]
0:00:00.029048794 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1013:color_matrix_debug: [0.000000 0.000000 0.000000
1.000000]
0:00:00.029066010 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1026:color_matrix_convert: [ 66 129 25 4096]
0:00:00.029080734 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1028:color_matrix_convert: [ -38 -74 112 32768]
0:00:00.029094831 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1030:color_matrix_convert: [ 112 -94 -18 32768]
0:00:00.029109267 73153 0x5650e5f6ea30 DEBUG video-converter
video-converter.c:1032:color_matrix_convert: [ 0 0 0 1]
```
Here I only seem to be able to see the NV12->GRAY8 conversion.
And I'm guessing the GRAY is just the pure Y extracted from the NV12.
But that results in a gray image where the blacks are grayish and if I look
at the histogram I can see I only have the limited range [16:235], which
explains the grayness.
I expected that here we also need to contrast stretch the gray image in
order to get back the full range [0:255].
So why the difference in how gstreamer threats the gray conversion depending
on the matrix conversion?
Is this a bug or is it because I misunderstood something in these standards?
--
Sent from: http://gstreamer-devel.966125.n4.nabble.com/
More information about the gstreamer-devel
mailing list