[RFC] Docs: drm: Move KMS properties table out to source files

[Graham Whaley]

On Tue, 2015-09-22 at 16:03 -0300, Danilo Cesar Lemes de Paula wrote:
> On 09/22/2015 07:22 AM, Graham Whaley wrote:
> > On Wed, Sep 02, 2015 at 02:50:52PM +0100, Graham Whaley wrote:
> > > > (RFC/test - not for merging)
> > > > The below is a test of moving the large HTML KMS properties
> > > > table
> > > > out
> > > > to markdown style in the appropriate files.
> > > > In the test we only use the first few rows of the existing KMS
> > > > table
> > > > an example.
> > > > We use a fixed width table as the other styles of table
> > > > supported
> > > > by
> > > > pandoc markdown do not support multi-column cells.
> > > > 
> > > > The test shows a couple of issues:
> > > >  1) double quote characters are being expanded in the fixed
> > > > width
> > > > table
> > > >  which then breaks the table alignment and leaves html style
> > > > <quote> tags
> > > >  in the text
> > 
> > Further to this item:
> >  Before I continue working on splitting the tables and converting
> > to
> > markdown (which btw Daniel does look feasible so far...), I thought
> > we
> > should understand what was going on with the markdown table quote
> > breakage.
> >  I think I know what is happenig.
> >  The problem revolves around highlight expansion in the kenrel-doc
> > script. In the output_highlight function we can see that first the
> > script does highlight expansion (with the eval of @dohighlight),
> > and
> > then it invokes the markdown processing.
> >  Done that way around what happens is:
> >  - @dohighlight expands any "X" to html/xml quote tag sequences
> >  - which can push the text beyond the table column widths,
> >  - and then the markdown tries to split the text over columns, and
> > manages to put the break in the middle of a tag, and thus breaks
> > the
> > tag formatting (see below)
> > 
> > At least that is what I think is happening.
> > I had thought maybe we could swap the markdown and highlight
> > processing
> > order, but then that presents a different issue - the markdown
> > table
> > contains metadata with quoted items (such as cols="5"), which the
> > highligh processing then expands into tags, and hence breaks the
> > table
> > format metadata.
> > 
> > As an example, using the following table !Pinclude'd into drm.tmpl:
> > /**
> >  * DOC: DRM generic
> >  *
> >  * : DRM generic properties
> >  *
> >  * +----------+--------+--------------------+---------+------------
> > ----
> > --------+
> >  * |Property  |Type    |Property Values     |Object  
> >  |Description/Restrictions|
> >  *
> > +==========+========+====================+=========+===============
> > ====
> > =====+
> >  * |"rotation"|BITMASK |{ 0, "rotate-0" }, {|CRTC,    |rotate
> > -(degrees)
> > rotates|
> >  * +----------+--------+--------------------+---------+------------
> > ----
> > --------+
> >  *
> >  */
> > 
> > The post-highlight pre-markdown text captured as the $orig_context
> > in
> > the markdown_to_docbook function is:
> > 
> > +----------+--------+--------------------+---------+---------------
> > ----
> > -----+
> > > Property  |Type    |Property Values     |Object  
> >  |Description/Restrictions|
> > +==========+========+====================+=========+===============
> > ====
> > =====+
> > > <quote>rotation</quote>|BITMASK |{ 0, <quote>rotate-0</quote> },
> > {|CRTC,    |rotate-(degrees) rotates|
> > +----------+--------+--------------------+---------+---------------
> > ----
> > -----+
> > 
> > which when processed (and you can do this by hand with 'pandoc -
> > -columns=80 -f markdown -t docbook ...' generates the broken tags
> > example:
> > 
> > ...
> > <tbody>
> >          <row>
> >            <entry>
> >              <para>
> >                <quote>rota
> >              </para>
> >            </entry>
> >            <entry>
> >              <para>
> >                tion</quo
> >              </para>
> >            </entry>
> >            <entry>
> >              <para>
> >                te>|BITMASK |{ 0, <qu
> >              </para>
> >            </entry>
> >            <entry>
> >              <para>
> >                ote>rotate
> >              </para>
> >            </entry>
> >            <entry>
> >              <para>
> >                -0</quote> }, {|CRTC, |rotate-(degrees) rotates
> >              </para>
> > ...
> > 
> > where you can see the quote tag processing has gone horribly wrong.
> > 
> > I believe we'll have the same problem for the other 'highlight'
> > processed items from kern-doc as well, meaning:
> >  funcname()
> >  $ENVVAR
> >  &struct_name
> >  @parameter
> >  %CONST
> >  
> > As the kern-doc processing has no knowledge of when it is about to
> > process a markdown table I can't think of an obvious way around
> > this.
> > At present I think the implicit rule is 'no highlight/expansion
> > items
> > allowed in markdown tables', which means all those quoted strings
> > for
> > the DRM properties cannot currently be migrated to look like
> > strings.
> > 
> > Danilo, or anybody, any ideas?
> 
> Your debugging is pretty precise. The "+=====+===" format is rigid. I
> don't know if it's a bug in pandoc or something from the spec itself.
> 
> May I suggest another format?
> 
> > Property   |  Type   |
> > ------------|:-------:|
> > "rotation" | BITMASK |
> 
> This format looks to work pretty well even if "rotation" is replaced
> by
> some very long string (which seems to be the problem). I didn't test
> it
> with kernel-doc, but pandoc looks to work fine with it.
> 
> I'm not sure if we can call it a proper fix, but it should allow you
> to
> keep the ball rolling.
> 
> --
> Danilo Cesar

Hi all,
 I had a look at the 'pipe' table type - and the other table types
available in pandoc_markdown. I've consolidated my thoughts into the
following analysis. Note, this thread is not presently on the docs-list
- we should pull that in as well if we think it may help.

Let's say there are 5 table types we can use in pandoc_markdown, if we
add in the use of the 'codeblock' type to the four formal
pandoc_markdown tables (see table later...).

My hypothesis' are:

1) We need to have a table format that supports multi-line rows, as
some rows in the DRM table are way too big to be putting on a single
line in the source file (for instance, the 'rotation' example I have
been using totals somewhere in the region of 349 characters as a single
line)
2) We need to support the kernel-doc highlight types in the tables as:
 2a) We need to use double quotes for strings in the DRM table at least
 2b) And I think the other highlight types will be useful in the future
3) We need a table type that allows the highlight tags to be
subsequently processed into the final document format (otherwise the
highlight tag expansions get left as raw tags in the final text).
4) Because (2) 'expands' a line width when it is processing/tagging the
highlights we need a table type that supports variable length lines,
and is not based on fixed width column processing.

If we summarise that in a table (ironically :-):

tabletype | multiline | variable | processed | suitable?
--------------------------------------------------------
simple    | No        | No       | Yes       | No
multi     | Yes       | No       | Yes       | No
grid      | Yes       | No       | Yes       | No
pipe      | No        | Yes      | Yes       | No
codeblock | Yes       | Yes      | No        | No

What we are looking for is a Yes in the first three columns. And thus,
presently it looks to me like none of the table formats are wholly
suitable :-(

I've still not thought of a way of tweaking the kernel-doc and pandoc
processing to work around this either, as they are done as different
passes/phases that neither has knowledge about the others
requirements.

As it stands, I'm failing to find a method to break out the DRM table
into markdown tables that I believe works, fundamentally due to this
'incompatibility' between the kernel-doc and pandoc_markdown processing
phases around the highlight processing.

 Graham