[PATCH 0/6] scripts/kernel-doc: Kernel-doc improvements

Mon Sep 14 05:11:00 PDT 2015

On 09/13/2015 05:58 PM, Daniel Vetter wrote:
> On Sun, Sep 13, 2015 at 9:13 PM, Jonathan Corbet <corbet at lwn.net> wrote:
>> On Sun, 13 Sep 2015 12:36:07 +0200
>> Daniel Vetter <daniel at ffwll.ch> wrote:
>>
>>> Personally I don't care which kind of text markup we pick and wich
>>> converter, as long as the project looks reasonable far away from
>>> immeninent death (way too many one-person projects on github in this
>>> area).

I wouldn't care either. I choose pandoc as it's old, kind of stable and
has a decent community around it, so it won't probably die due the lack
of interested developers.

That said, cmark is not a option (at least for now). With the current
approach we need a tool capable of reading markdown text and output
DocBook tags, as we're actually still using docbook to spit all kinds of
Documentation formats (html, man, pdf). Looks like cmark can only do
markdown->html.

I did look into other options though, but they were all mostly one-man
job. But yeah, basically any tool capable of converting
markdown->docbook will work. I'm always listening to suggestions =)

>>>
>>> But if we have this discussion I'd like to decouple it from the other
>>> kerneldoc improvemnts in this series (patches 1, 5 and 6). If we cant
>>> agree on the text markup then drm docs will simply look a bit funny for
>>> everyone else. But if the inline struct stuff won't happen 0-day will
>>> scream around (and there's already patches which use the new layout).
>>
>> Those patches are:
>>
>> 1)  scripts/kernel-doc: Replacing highlights hash by an array
>> 5)  scripts/kernel-doc: Improve Markdown results
>> 6)  scripts/kernel-doc: Processing -nofunc for functions only
>>
>> #1 is fine, I'll merge that today.  #6 is already merged.  #5 is a markdown
>> patch, though, and doesn't make sense without the others?  Are you thinking
>> about #3 (drm/doc: Convert to markdown)?  That one would almost work (it
>> depends on #2 currently) and it nicely shows *why* I'd like to get away from
>> XML...

#5 is the DRM markdown conversion, which depends on #2 and #4. So if a
decision about this is going during KS, we need to hold those three patches.

> 
> Sorry I mixed things up, #5 is ok to leave out. I thought about
> "scripts/kernel-doc Allow struct arguments documentation in struct
> body" but you already merged that one for 4.2. Which I missed, but
> which is great since that's the one that would cause the big
> conflicts.
> 
>>>> 2 We're constructing an increasingly complicated document-processing
>>>>   mechanism with a lot of independently moving parts.  What if we
>>>>   converted the whole document to markdown and dispensed with the XML
>>>>   part altogether?  Making the source files simpler and dispensing with
>>>>   the xmlto requirement would be a big win, IMO.
>>>
>>> Who's going to convert the almost 30kloc of xml template (which often have
>>> large amounts of texts) and the over 60k kerneldoc comments that we have
>>> already?
>>
>> I thought you'd do that :)
>>
>> Seriously, though, a change would be a big job.  There's a reason I've said
>> several times that it would make no sense to delay the work at hand in the
>> hopes of somebody doing this whole job instead.  But we can certainly
>> ponder what might be better.
>>
>> Getting rid of the XML stuff may well simplify the whole process and make
>> the documentation much more accessible for those who would change it; that
>> could be a goal worth going for.  Oh, and anything requiring changes to the
>> kerneldoc comments is going nowhere, that was never something I'd
>> contemplated.  Those comments are fine.
> 
> Well my goal is to be able to have all the programmer docs in the
> code, including any high-level overview sections to tie everything
> together (hence hyperlinks and better formatting). But then you still
> need some skeleton to make a coherent whole out of all the parts, and
> I think the docbook xml templates we have serve rather fine for that
> role. Writing text in xml is indeed horrid, but if we can create
> in-line DOC: sections with decent formatting there's really no need
> for that. From that angle this work here already has the goal to get
> rid of the xml - I plane to move all the existing text in the drm.tmpl
> xml into inline DOC: kerneldoc commets.
> 
> And at least gtkdoc (which we use extensively for the userpace
> test/tools repo) relies on some xml thing to tie different topics
> together too. So that seems pretty standard.
> 
>> But any such change needs a lot of thought and a reasonable proof of
>> concept.  Meanwhile we have work that can make the docs better now, and I
>> want to merge it.  But we should choose the tools we use carefully, and I
>> anticipate a lot of opposition to one that has to drag in 70 Haskell
>> packages with it.
> 
> Well personally I don't care about the exact tooling, as long as:
> - it's a reasonable alive project
> - packages available on distros (works for me on both debian and fedora)
> - we can muck around with how things are integrated into overall docs
> (which is the case since pandoc is only used as a paragraph formatter
> here).
> 
> The last one is something we stumbled over for the userspace side
> where we wanted to have better tooling for documenting testcases.
> Upstream gtkdoc is receptive, but getting anything in takes forever
> and then there's the 6 month release schedule too. And there's also
> the problem that we can't change the kerneldoc grammar (e.g. gtkdoc
> has the code comment parser integrated and ofc it has slightly
> different rules to mark up references to structs/enums).
> 
> I'm also trying to build up a doc team here with a full-time tech
> writer for drm/i915 topics. So I'm very much interested in keeping
> this all alive. But I don't have the time to mass-convert everything
> else, and I'm also not sure it's a good idea: In my experience all the
> doc toolchain suck somewhat, and it's better to be consistent within a
> project instead of proliferating different solutions. It's hard enough
> already to get developers to write docs and even harder to find
> someone who writes good docs ...
> 
> Ofc we already have a big split between the ad-hoc text files in
> Documentation and the docbook stuff in Documentation/DocBook. Otoh
> docbook is the only way to extract the kerneldoc comments into
> something readable and indexed.
> 
>>>> I will not make #2 be a precondition to getting some form of this work
>>>> merged, but I would like to have a good answer for #1.  Adding such a
>>>> heavyweight dependency (even as an optional one) needs to have a pretty
>>>> good story behind it.
>>>
>>> Should we discuss #1 at ks? Imo as long as no one pipes up to do the
>>> massive conversion away from the current toolchain (and subsystem
>>> maintainers won't just kill that effort because it causes too much churn)
>>> moving forward with the kerneldoc toolchain we have makes sense. Hence I'd
>>> like to see those patches landed before we resolve #1 if possible.
>>
>> I've tossed the topic out there; the kernel summit agenda hasn't been made
>> yet, though.  In any case, I'd be happy to resolve this particular issue
>> before then if it were possible.
> 
> I don't mind much if we delay this part until KS. All the text markup
> styles are pretty close. So fixing up one dialect for another for the
> few patches I'll likely pull in for 4.4 won't be a lot of work if we
> decide to go with something else. Or even if we decide to switch
> markup styles much later, it should all be doable with a few sed jobs.
> 
> Only problem I see is that thus far no one else seems to care much,
> and if we change the approach I need to convince management here again
> that we need to invest some time to make it happen (and some excuses
> why this approach here wasn't ok). That will probably suck down more
> time than converting to whatever we decide on. And like I said I
> personally don't care one bit what the toolchain looks like or the
> exact flavour of text markup we pick.
> 
> Cheers, Daniel
> 