[Bug 707361] video: Add support for 64x32 tiled NV12 color format

[GStreamer (bugzilla.gnome.org)]

https://bugzilla.gnome.org/show_bug.cgi?id=707361
  GStreamer | gst-plugins-base | git

--- Comment #12 from Nicolas Dufresne <nicolas.dufresne at collabora.co.uk> 2013-11-17 02:34:58 UTC ---
After a long discussion on IRC, it was revealed that GstVideoInfo width and
height are cropped width and height, while the layout in this format depends on
the image size (more precisely, original number or rows and columns). This
information does not exist in GStreamer and cannot be deduced (the crop meta
consist of x, y, width, height where width and height will likely have to match
the VideoInfo width and height). Note that this cropping API matches V4L2 API,
but not Android API (which uses crop-left, crop-right, crop-top and
crop-bottom).

Take note though that videoconvert does not support incoming buffer with crop
meta at the moment, and destination buffer is assumed to have no cropping (no x
parameter to pack). But on the other hand, "standard" formats can be cropped on
right and bottom without having to do anything, thus a buffer could be somehow
cropped by simply reducing the width and the height. This is also true for this
format, as long as the image width and height is preserved somewhere.

Why does it matter for NV12, it's because the pattern used to layout the tiles
has a special case on the last row. If it is an odd row, the tiles are placed
linearly instead of the Z pattern. Obviously, they do so since otherwise they'd
have to wast a row of tiles, which is a lot bigger then a row of pixels. In the
end, if height is cropped more then a tile height, we need to know what was the
original height otherwise we will miss-interpret the last line.

This being said, we need not to confused bugs with problems, there is three
known bugs in the code atm:

Converter Code:
1. Number of columns shall be determine using stride (rounded down to multiple
of 128), not width rounded up. Obviously, ROUND_DOWN_128 (sride) >=
ROUND_UP_128 (width). This is defined by the format, and most of the time this
will be the same.
2. The tile_position() function assumed stride being a multiple of 128, this is
always true in general, but we can improve this to support any random stride
without affecting much the performance.
3. The unpack() function ignores the x. This is a bit tricky to handle, but
right now we assume it's always 0, which will probably remain true until we
start supporting incoming buffers with crop meta. (I can fix that, it's just
about adding more special cases, i.e. if x is odd, I simply did not notice the
signature difference).

Now for the missing information, which is not defined in GStreamer, it was
suggested to add an extra plane with the extra parameters. I first thought that
adding the full index would be nice, but adding memory offset in a plane seems
a bit dangerous in the end. Also, I felt that the most generic and correct way
to do this would be to double the planes, ensuring that each planes has it's
map, and that the map has it's own stride. Though this is not possible since it
would require more then the maximum of 4 planes we are allowed. (8 would be a
number that fits all format).

In the end, I will encode the image height in the place (in big endian). It is
very generic, not dedicated to tiles, and could be reused for wide variety of
format with special layout, and where this layout can be expressed by a
function.

The second question was where to setup that plane, the answer is where the data
is created. As this information is not stored in GStreamer, the last time we
have this information is when we assemble the buffer. This is very similar to
how palettes are handled, so this is how the extra parameter will be handled.
The only problem with that approach is that we will not be able to support the
same thing on top of A420, as there would be not enough planes left in
Gstreamer API.

We might need a fast path for Android use case, though I think that the sink
should be able to support that format directly if the decoder produces it. If
Andoni could clarify this, adding a de-tiling fast path should be rather easy
but require some time and I'd be incline not to do that now. For me that
converter is simply for enable videotestsrc to produce this format.

-- 
Configure bugmail: https://bugzilla.gnome.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.
You are the assignee for the bug.