[Mesa-dev] Documenting with Sphinx

[Jani Nikula]

On Wed, 14 Dec 2016, Robert Bragg <robert at sixbynine.org> wrote:
> On Tue, Dec 13, 2016 at 11:08 PM, Rob Clark <robdclark at gmail.com> wrote:
>> On Tue, Dec 13, 2016 at 5:47 PM, Eric Anholt <eric at anholt.net> wrote:
>>> Jason Ekstrand <jason at jlekstrand.net> writes:
>>>
>>>> Hey All,
>>>> I don't figure this will be terribly controversial (I'm about to be wrong,
>>>> aren't I?) but how do people feel about switching our "primary"
>>>> documentation focus, as far as we have one, to sphinx?  Right now, Gallium
>>>> uses sphinx for documenting all of it's "public" interfaces.  A while ago,
>>>> Connor wrote some nice NIR documentation using sphinx that never got
>>>> merged.  I've also spent some time this year and written some additional
>>>> documentation about Intel surface layout that I'd like to have a place to
>>>> put.  The thing that's in common with each of these is that they contain
>>>> quite a bit of prose about the general theory of how things work and not
>>>> just code comments so sphinx seems like a fairly good fit.  There are also
>>>> some sort-of "architectural" things in the Vulkan driver that I would like
>>>> to document better and we don't have a good place for that right now.
>>>>
>>>> Here's what I have in mind:
>>>>
>>>> Each component that we care to document (this may not be everything!) would
>>>> have a docs/ or sphinx/ folder.  In that folder would be any side-band (not
>>>> in code comments) documentation.  We would also set up some sort of
>>>> source-scraping tool to turn inline comments into sphinx documentation.  At
>>>> the component maintainer's discretion, that code could either be imported
>>>> into sphinx whole-sale or curated by importing bits into other
>>>> documentation files in a more carefully curated manner.  Of course, a
>>>> maintainer could also choose to do all of their documentation side-band
>>>> (sphinx allows for this) or to not document at all. :-)
>>
>> So, +100 to sphinx... I've been quite happy with it (and would be nice
>> to merge the NIR docs too while where at it).  Maybe someday we'll
>> have a "mesa book" after all ;-)
>>
>>
>>> I've written doxygen-style comments in my code for a long time, but
>>> never really looked at the doxygen output, so I wouldn't say I'm wedded
>>> to doxygen.  I think some side-band docs for architectural descriptions
>>> (like NIR needs) would be great, and if sphinx would help with
>>> encouraging those to be written and merged, then that's pretty good with
>>> me.
>>>
>>> I'm a little skeptical of sphinx due to not being able to extract docs
>>> From C comments in code in the base tool.  Apparently Breathe can chain
>>> doxygen into sphinx, so hopefully a "let's get excited about sphinx"
>>> patch series would convert our doxygen-based docs generation to using
>>> that.
>>
>> So as far as I understand on kernel side we are *somehow* extracting
>> out C comment docs and combining w/ sphinx stuff..  maybe there is
>> something to re-use from that.
>
> The actual parsing, extraction and generation of reStructureText for
> the kernel is handled by scripts/kernel-doc, which is a perl script
> based around regex parsing of C code. Jani Nikula has also written a
> sphinx extension for using this script which can be seen in
> Documentation/sphinx/. As you can probably guess kernel-doc is pretty
> linux kernel specific. The script existed before the recent adoption
> of sphinx and has support for docbook and man page generation. Being
> based on regex parsing it has a somewhat limited understanding of the
> underlying types of symbols being documented.
>
> I recently raised the idea of using clang's python bindings to extract
> kerneldoc on dri-devel due to some of the limitations of kernel-doc I
> was hitting and pointed at an experimental tool I wrote last year for
> generating reStructureText based on gtk-doc comments:
> https://github.com/rib/clib/blob/master/site/rst-from-c.py Probably
> not developed enough to be very useful here, but provides one basic
> example of generating reStructuredText from structured comments.
>
> Jani mentioned he had a bit of a side project going to use the same
> clang bindings to support extracting doxygen comments, so maybe what
> he's been working on could be useful for Mesa too, but I think maybe
> that was for doxygen itself so maybe nothing to do with generating
> reStructureText.

I'm afraid what I have is really not ready for consumption by
anyone. The idea is to basically have a Sphinx extension to extract
documentation comments from source using python-clang. The goal is to
keep it *very* simple, and *not* do any of the bloat Doxygen has. If you
want to bolt stuff into Sphinx, why have the complexity of Doxygen, and
the format conversion with inevitable hickups, when the documentation
comments could just be reStructureText to begin with?

I haven't pursued this for the kernel, because I doubt adding a clang
dependency on the documentation build would go very well. Plus you'd
have to emulate everything kernel-doc the perl script monster does, and
I kind of want to do something that sucks less.

> In the specific case of doxygen format comments, I have a vague
> recollection that clang itself can actually parse and provide some
> kind of AST for your comments, so maybe that would be convenient.

The AST does include comments right above each cursor, but it falls
short for comments not attached to cursors. You have to walk the tokens
for that. I think I have most of this solved, though it hasn't seen much
testing.

No matter what, I think you should pursue documentation comment
extraction. Manually documenting C APIs and stuff in Sphinx is not
fun. If you want to keep compatibility with Doxygen, you should probably
look into Doxygen+Breathe, but be wary of the format
conversions. Otherwise, I think the options are to reuse what kernel has
(a perl script nobody in their right mind admits to understanding) or
write a Sphinx extension, likely based on python-clang (which is not
going to be very feature rich or Doxygen compatible at the beginning,
even if I polish what I have to give you a tiny jump start).

I'm not subscribed on mesa-dev, but send mail my way if you want my
opinions on whatever you're planning.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center