[Mesa-dev] Documenting with Sphinx

[Robert Bragg]

On Wed, Dec 14, 2016 at 7:17 PM, Jani Nikula
<jani.nikula at linux.intel.com> wrote:
> On Wed, 14 Dec 2016, Robert Bragg <robert at sixbynine.org> wrote:
>> On Tue, Dec 13, 2016 at 11:08 PM, Rob Clark <robdclark at gmail.com> wrote:
>>> On Tue, Dec 13, 2016 at 5:47 PM, Eric Anholt <eric at anholt.net> wrote:
>>>> Jason Ekstrand <jason at jlekstrand.net> writes:
>>>>
>>>>> Hey All,
>>>>> I don't figure this will be terribly controversial (I'm about to be wrong,
>>>>> aren't I?) but how do people feel about switching our "primary"
>>>>> documentation focus, as far as we have one, to sphinx?  Right now, Gallium
>>>>> uses sphinx for documenting all of it's "public" interfaces.  A while ago,
>>>>> Connor wrote some nice NIR documentation using sphinx that never got
>>>>> merged.  I've also spent some time this year and written some additional
>>>>> documentation about Intel surface layout that I'd like to have a place to
>>>>> put.  The thing that's in common with each of these is that they contain
>>>>> quite a bit of prose about the general theory of how things work and not
>>>>> just code comments so sphinx seems like a fairly good fit.  There are also
>>>>> some sort-of "architectural" things in the Vulkan driver that I would like
>>>>> to document better and we don't have a good place for that right now.
>>>>>
>>>>> Here's what I have in mind:
>>>>>
>>>>> Each component that we care to document (this may not be everything!) would
>>>>> have a docs/ or sphinx/ folder.  In that folder would be any side-band (not
>>>>> in code comments) documentation.  We would also set up some sort of
>>>>> source-scraping tool to turn inline comments into sphinx documentation.  At
>>>>> the component maintainer's discretion, that code could either be imported
>>>>> into sphinx whole-sale or curated by importing bits into other
>>>>> documentation files in a more carefully curated manner.  Of course, a
>>>>> maintainer could also choose to do all of their documentation side-band
>>>>> (sphinx allows for this) or to not document at all. :-)
>>>
>>> So, +100 to sphinx... I've been quite happy with it (and would be nice
>>> to merge the NIR docs too while where at it).  Maybe someday we'll
>>> have a "mesa book" after all ;-)
>>>
>>>
>>>> I've written doxygen-style comments in my code for a long time, but
>>>> never really looked at the doxygen output, so I wouldn't say I'm wedded
>>>> to doxygen.  I think some side-band docs for architectural descriptions
>>>> (like NIR needs) would be great, and if sphinx would help with
>>>> encouraging those to be written and merged, then that's pretty good with
>>>> me.
>>>>
>>>> I'm a little skeptical of sphinx due to not being able to extract docs
>>>> From C comments in code in the base tool.  Apparently Breathe can chain
>>>> doxygen into sphinx, so hopefully a "let's get excited about sphinx"
>>>> patch series would convert our doxygen-based docs generation to using
>>>> that.
>>>
>>> So as far as I understand on kernel side we are *somehow* extracting
>>> out C comment docs and combining w/ sphinx stuff..  maybe there is
>>> something to re-use from that.
>>
>> The actual parsing, extraction and generation of reStructureText for
>> the kernel is handled by scripts/kernel-doc, which is a perl script
>> based around regex parsing of C code. Jani Nikula has also written a
>> sphinx extension for using this script which can be seen in
>> Documentation/sphinx/. As you can probably guess kernel-doc is pretty
>> linux kernel specific. The script existed before the recent adoption
>> of sphinx and has support for docbook and man page generation. Being
>> based on regex parsing it has a somewhat limited understanding of the
>> underlying types of symbols being documented.
>>
>> I recently raised the idea of using clang's python bindings to extract
>> kerneldoc on dri-devel due to some of the limitations of kernel-doc I
>> was hitting and pointed at an experimental tool I wrote last year for
>> generating reStructureText based on gtk-doc comments:
>> https://github.com/rib/clib/blob/master/site/rst-from-c.py Probably
>> not developed enough to be very useful here, but provides one basic
>> example of generating reStructuredText from structured comments.
>>
>> Jani mentioned he had a bit of a side project going to use the same
>> clang bindings to support extracting doxygen comments, so maybe what
>> he's been working on could be useful for Mesa too, but I think maybe
>> that was for doxygen itself so maybe nothing to do with generating
>> reStructureText.
>
> I'm afraid what I have is really not ready for consumption by
> anyone. The idea is to basically have a Sphinx extension to extract
> documentation comments from source using python-clang. The goal is to
> keep it *very* simple, and *not* do any of the bloat Doxygen has. If you
> want to bolt stuff into Sphinx, why have the complexity of Doxygen, and
> the format conversion with inevitable hickups, when the documentation
> comments could just be reStructureText to begin with?
>
> I haven't pursued this for the kernel, because I doubt adding a clang
> dependency on the documentation build would go very well. Plus you'd
> have to emulate everything kernel-doc the perl script monster does, and
> I kind of want to do something that sucks less.
>
>> In the specific case of doxygen format comments, I have a vague
>> recollection that clang itself can actually parse and provide some
>> kind of AST for your comments, so maybe that would be convenient.
>
> The AST does include comments right above each cursor, but it falls
> short for comments not attached to cursors. You have to walk the tokens
> for that. I think I have most of this solved, though it hasn't seen much
> testing.

Right, what I was thinking of though is that clang itself can actually
parse doxygen style comment sections like \brief, \param and \returns:

http://clang.llvm.org/doxygen/classclang_1_1comments_1_1Parser.html

Seems a bit mad, but this e.g. helps enable a -Wdocumentation warning
to check \params match actual function arguments

I was wondering if that can somehow be exposed via the AST accessible
through the python bindings.

Regards,
- Robert

>
> No matter what, I think you should pursue documentation comment
> extraction. Manually documenting C APIs and stuff in Sphinx is not
> fun. If you want to keep compatibility with Doxygen, you should probably
> look into Doxygen+Breathe, but be wary of the format
> conversions. Otherwise, I think the options are to reuse what kernel has
> (a perl script nobody in their right mind admits to understanding) or
> write a Sphinx extension, likely based on python-clang (which is not
> going to be very feature rich or Doxygen compatible at the beginning,
> even if I polish what I have to give you a tiny jump start).
>
> I'm not subscribed on mesa-dev, but send mail my way if you want my
> opinions on whatever you're planning.
>
> BR,
> Jani.
>
>
> --
> Jani Nikula, Intel Open Source Technology Center