[PATCH 2/6] docs-rst: automatically convert Graphviz and SVG images

Daniel Vetter daniel.vetter at ffwll.ch
Tue Feb 28 17:13:15 UTC 2017


From: Markus Heiser <markus.heiser at darmarit.de>

This patch brings scalable figure, image handling and a concept to
embed *render* markups:

* DOT (http://www.graphviz.org)
* SVG

For image handling use the 'image' replacement::

    .. kernel-image::  svg_image.svg
       :alt:    simple SVG image

For figure handling use the 'figure' replacement::

    .. kernel-figure::  svg_image.svg
       :alt:    simple SVG image

       SVG image example

Embed *render* markups (or languages) like Graphviz's **DOT** is
provided by the *render* directive.::

  .. kernel-render:: DOT
     :alt: foobar digraph
     :caption: Embedded **DOT** (Graphviz) code.

     digraph foo {
      "bar" -> "baz";
     }

The *render* directive is a concept to integrate *render* markups and
languages, yet supported markups:

* DOT: render embedded Graphviz's **DOT**
* SVG: render embedded Scalable Vector Graphics (**SVG**)

v2: s/DOC/DOT/ in a few places (by Daniel).

v3: Simplify stuff a bit (by Daniel):

- Remove path detection and setup/check code for that. In
  Documentation/media/Makefile we already simply use these tools,
  better to have one consolidated check if we want/need one. Also
  remove the convertsvg support, we require ImageMagick's convert
  already in the doc build, no need for a 2nd fallback.

- Use sphinx for depency tracking, remove hand-rolled version.

- Forward stderr from dot and convert, otherwise debugging issues with
  the diagrams is impossible.

Cc: Jonathan Corbet <corbet at lwn.net>
Cc: linux-doc at vger.kernel.org
Cc: Jani Nikula <jani.nikula at linux.intel.com>
Cc: Mauro Carvalho Chehab <mchehab at s-opensource.com>
Signed-off-by: Markus Heiser <markus.heiser at darmarit.de> (v1)
Signed-off-by: Daniel Vetter <daniel.vetter at intel.com>
---
 Documentation/conf.py                 |   2 +-
 Documentation/doc-guide/hello.dot     |   3 +
 Documentation/doc-guide/sphinx.rst    |  90 ++++++-
 Documentation/doc-guide/svg_image.svg |  10 +
 Documentation/process/changes.rst     |   7 +-
 Documentation/sphinx/kfigure.py       | 442 ++++++++++++++++++++++++++++++++++
 6 files changed, 548 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/doc-guide/hello.dot
 create mode 100644 Documentation/doc-guide/svg_image.svg
 create mode 100644 Documentation/sphinx/kfigure.py

diff --git a/Documentation/conf.py b/Documentation/conf.py
index f6823cf01275..e3f537ce2935 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -34,7 +34,7 @@ from load_config import loadConfig
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain']
+extensions = ['kerneldoc', 'rstFlatTable', 'kernel_include', 'cdomain', 'kfigure']
 
 # The name of the math extension changed on Sphinx 1.4
 if major == 1 and minor > 3:
diff --git a/Documentation/doc-guide/hello.dot b/Documentation/doc-guide/hello.dot
new file mode 100644
index 000000000000..504621dfc595
--- /dev/null
+++ b/Documentation/doc-guide/hello.dot
@@ -0,0 +1,3 @@
+graph G {
+      Hello -- World
+}
diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
index 532d65b70500..b902744ce7dd 100644
--- a/Documentation/doc-guide/sphinx.rst
+++ b/Documentation/doc-guide/sphinx.rst
@@ -34,8 +34,10 @@ format-specific subdirectories under ``Documentation/output``.
 
 To generate documentation, Sphinx (``sphinx-build``) must obviously be
 installed. For prettier HTML output, the Read the Docs Sphinx theme
-(``sphinx_rtd_theme``) is used if available. For PDF output, ``rst2pdf`` is also
-needed. All of these are widely available and packaged in distributions.
+(``sphinx_rtd_theme``) is used if available. For PDF output you'll also need
+``XeLaTeX`` and CairoSVG (http://cairosvg.org) or alternatively ``convert(1)``
+from ImageMagick (https://www.imagemagick.org). All of these are widely
+available and packaged in distributions.
 
 To pass extra options to Sphinx, you can use the ``SPHINXOPTS`` make
 variable. For example, use ``make SPHINXOPTS=-v htmldocs`` to get more verbose
@@ -232,3 +234,87 @@ Rendered as:
       * .. _`last row`:
 
         - column 3
+
+
+Figures & Images
+================
+
+If you want to add an image, you should use the ``kernel-figure`` and
+``kernel-image`` directives. E.g. to insert a figure with a scalable
+image format use SVG::
+
+    .. kernel-figure::  svg_image.svg
+       :alt:    simple SVG image
+
+       SVG image example
+
+.. kernel-figure::  svg_image.svg
+   :alt:    simple SVG image
+
+   SVG image example
+
+The kernel figure (and image) directive support **DOT** formated files, see
+
+* DOT: http://graphviz.org/pdf/dotguide.pdf
+* Graphviz: http://www.graphviz.org/content/dot-language
+
+A simple example::
+
+  .. kernel-figure::  hello.dot
+     :alt:    hello world
+
+     DOT's hello world example
+
+.. kernel-figure::  hello.dot
+   :alt:    hello world
+
+   DOT's hello world example
+
+Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the
+``kernel-render`` directives.::
+
+  .. kernel-render:: DOT
+     :alt: foobar digraph
+     :caption: Embedded **DOT** (Graphviz) code.
+
+     digraph foo {
+      "bar" -> "baz";
+     }
+
+How this will be rendered depends on the installed tools. If Graphviz is
+installed, you will see an vector image. If not the raw markup is inserted as
+*literal-block*.
+
+.. kernel-render:: DOT
+   :alt: foobar digraph
+   :caption: Embedded **DOT** (Graphviz) code.
+
+   digraph foo {
+      "bar" -> "baz";
+   }
+
+The *render* directive has all the options known from the *figure* directive,
+plus option ``caption``.  If ``caption`` has a value, a *figure* node is
+inserted. If not, a *image* node is inserted.
+
+Embedded **SVG**::
+
+  .. kernel-render:: SVG
+     :caption: Embedded **SVG** markup.
+     :alt: so-nw-arrow
+
+     <?xml version="1.0" encoding="UTF-8"?>
+     <svg xmlns="http://www.w3.org/2000/svg" version="1.1" ...>
+        ...
+     </svg>
+
+.. kernel-render:: SVG
+   :caption: Embedded **SVG** markup.
+   :alt: so-nw-arrow
+
+   <?xml version="1.0" encoding="UTF-8"?>
+   <svg xmlns="http://www.w3.org/2000/svg"
+     version="1.1" baseProfile="full" width="70px" height="40px" viewBox="0 0 700 400">
+   <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/>
+   <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/>
+   </svg>
diff --git a/Documentation/doc-guide/svg_image.svg b/Documentation/doc-guide/svg_image.svg
new file mode 100644
index 000000000000..5405f85b8137
--- /dev/null
+++ b/Documentation/doc-guide/svg_image.svg
@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- originate: https://commons.wikimedia.org/wiki/File:Variable_Resistor.svg -->
+<svg xmlns="http://www.w3.org/2000/svg"
+	version="1.1" baseProfile="full"
+	width="70px" height="40px" viewBox="0 0 700 400">
+	<line x1="0" y1="200" x2="700" y2="200" stroke="black" stroke-width="20px"/>
+	<rect x="100" y="100" width="500" height="200" fill="white" stroke="black" stroke-width="20px"/>
+	<line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/>
+	<polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/>
+</svg>
diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 56ce66114665..e4f25038ef65 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -318,9 +318,10 @@ PDF outputs, it is recommended to use version 1.4.6.
 .. note::
 
   Please notice that, for PDF and LaTeX output, you'll also need ``XeLaTeX``
-  version 3.14159265. Depending on the distribution, you may also need
-  to install a series of ``texlive`` packages that provide the minimal
-  set of functionalities required for ``XeLaTex`` to work.
+  version 3.14159265. Depending on the distribution, you may also need to
+  install a series of ``texlive`` packages that provide the minimal set of
+  functionalities required for ``XeLaTex`` to work. For PDF output you'll also
+  need ``convert(1)`` from ImageMagick (https://www.imagemagick.org).
 
 Other tools
 -----------
diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
new file mode 100644
index 000000000000..8f749d389dae
--- /dev/null
+++ b/Documentation/sphinx/kfigure.py
@@ -0,0 +1,442 @@
+# -*- coding: utf-8; mode: python -*-
+# pylint: disable=C0103
+u"""
+    scalable figure and image handling
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    Sphinx extension which implements scalable image handling.
+
+    :copyright:  Copyright (C) 2016  Markus Heiser
+    :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
+
+    The build for image formats depence on image's source format and output's
+    destination format. This extension implement methods to simplify image
+    handling from the author's POV. Directives like ``kernel-figure`` implement
+    methods *to* always get the best output-format even if some tools are not
+    installed.For more details take a look at ``convert_image(...)`` which is
+    the core of all conversions.
+
+    * ``.. kernel-image``: for image handling / ``.. image::`` replacement
+
+    * ``.. kernel-figure``: for figure handling / ``.. figure::`` replacement
+
+    * ``.. kernel-render``: for render markup / a concept to embed *render*
+      markups (or languages). Supported markups (see ``RENDER_MARKUP_EXT``)
+
+      + ``DOT``: render embedded Graphviz's **DOC**
+      + ``SVG``: render embedded Scalable Vector Graphics (**SVG**)
+      + ... *developable*
+
+    Used tools:
+
+    * ``dot(1)``: Graphviz (http://www.graphviz.org). If Graphviz is not
+      available, the DOT language is inserted as literal-block.
+
+    * SVG to PDF: To generate PDF, you need at least one of this tools:
+
+      - ``convert(1)``: ImageMagick (https://www.imagemagick.org)
+
+    List of customizations:
+
+    * generate PDF from SVG / used by PDF (LaTeX) builder
+
+    * generate SVG (html-builder) and PDF (latex-builder) from DOT files.
+      DOT: see http://www.graphviz.org/content/dot-language
+
+    """
+
+import os
+from os import path
+import subprocess
+from hashlib import sha1
+import sys
+
+from docutils import nodes
+from docutils.statemachine import ViewList
+from docutils.parsers.rst import directives
+from docutils.parsers.rst.directives import images
+
+from sphinx.directives import patches
+
+__version__  = '1.0'
+
+# simple helper
+# -------------
+
+def mkdir(folder, mode=0o775):
+    if not path.isdir(folder):
+        os.makedirs(folder, mode)
+
+# def debug_handle(self, node):           # pylint: disable=W0613
+#     from linuxdoc.kernel_doc import CONSOLE
+#     CONSOLE()
+
+def pass_handle(self, node):           # pylint: disable=W0613
+    pass
+
+# setup conversion tools and sphinx extension
+# -------------------------------------------
+
+def setup(app):
+    # image handling
+    app.add_directive("kernel-image",  KernelImage)
+    app.add_node(kernel_image,
+                 html    = (visit_kernel_image, pass_handle),
+                 latex   = (visit_kernel_image, pass_handle),
+                 texinfo = (visit_kernel_image, pass_handle),
+                 text    = (visit_kernel_image, pass_handle),
+                 man     = (visit_kernel_image, pass_handle), )
+
+    # figure handling
+    app.add_directive("kernel-figure", KernelFigure)
+    app.add_node(kernel_figure,
+                 html    = (visit_kernel_figure, pass_handle),
+                 latex   = (visit_kernel_figure, pass_handle),
+                 texinfo = (visit_kernel_figure, pass_handle),
+                 text    = (visit_kernel_figure, pass_handle),
+                 man     = (visit_kernel_figure, pass_handle), )
+
+    # render handling
+    app.add_directive('kernel-render', KernelRender)
+    app.add_node(kernel_render,
+                 html    = (visit_kernel_render, pass_handle),
+                 latex   = (visit_kernel_render, pass_handle),
+                 texinfo = (visit_kernel_render, pass_handle),
+                 text    = (visit_kernel_render, pass_handle),
+                 man     = (visit_kernel_render, pass_handle), )
+
+    return dict(
+        version = __version__,
+        parallel_read_safe = True,
+        parallel_write_safe = True
+    )
+
+# integrate conversion tools
+# --------------------------
+
+RENDER_MARKUP_EXT = {
+    # The '.ext' must be handled by convert_image(..) function's *in_ext* input.
+    # <name> : <.ext>
+    'DOT'   : '.dot'
+    , 'SVG' : '.svg'
+}
+
+def convert_image(img_node, translator):  # pylint: disable=R0912
+    """Convert an image node for the builder.
+
+    Different builder prefer different image formats, e.g. *latex* builder
+    prefer PDF while *html* builder prefer SVG format for images.
+
+    This function handles outputs image formats in depence of source the format
+    of the image and the translator's output format. This also means to
+    manipulate/update the *image* dictionary of the builder (``builder.images``)
+
+    """
+    fname, in_ext = path.splitext(path.basename(img_node['uri']))
+    src_fname     = path.join(translator.builder.srcdir, img_node['uri'])
+    src_folder    = path.dirname(img_node['uri'])
+    out_dir       = translator.builder.outdir
+    dst_fname     = None
+
+    # in kernel builds, use 'make SPHINXOPTS=-v' to see verbose messages
+    verbose = translator.builder.app.verbose
+    warn    = translator.builder.warn
+
+    verbose('assert best format for: ' + img_node['uri'])
+
+    if in_ext == '.dot':
+
+        # ----------
+        # handle DOT
+        # ----------
+
+        dst_fname = path.join(out_dir, fname + '.pdf')
+
+        if translator.builder.format == 'html':
+            dst_fname = path.join(out_dir, src_folder, fname + '.svg')
+        else:
+            # all other builder formats will include DOT as raw
+            with open(src_fname, "r") as dot:
+                data = dot.read()
+                node = nodes.literal_block(data, data)
+                img_node.replace_self(node)
+
+
+    elif in_ext == '.svg':
+
+        # ----------
+        # handle SVG
+        # ----------
+
+        if translator.builder.format == 'latex':
+            dst_fname = path.join(out_dir, fname + '.pdf')
+
+    if dst_fname:
+        name = dst_fname[len(out_dir) + 1:]
+        # the builder needs not to copy one more time, so pop it if exists.
+        translator.builder.images.pop(img_node['uri'], None)
+        img_node['uri'] = dst_fname
+        img_node['candidates'] = {'*': dst_fname}
+
+        mkdir(path.dirname(dst_fname))
+
+        if in_ext == '.dot':
+            verbose('convert DOT to: {out}/' + name)
+            dot2format(src_fname, dst_fname)
+
+        elif in_ext == '.svg':
+            verbose('convert SVG to: {out}/' + name)
+            svg2pdf(src_fname, dst_fname)
+
+def dot2format(dot_fname, out_fname):
+    """Converts DOT file to ``out_fname`` using ``dot(1)``.
+
+    * ``dot_fname`` pathname of the input DOT file, including extension ``.dot``
+    * ``out_fname`` pathname of the output file, including format extension
+
+    The *format extension* depends on the ``dot`` command (see ``man dot``
+    option ``-Txxx``). Normally you will use one of the following extensions:
+
+    - ``.ps`` for PostScript,
+    - ``.svg`` or ``svgz`` for Structured Vector Graphics,
+    - ``.fig`` for XFIG graphics and
+    - ``.png`` or ``gif`` for common bitmap graphics.
+
+    """
+    out_format = path.splitext(out_fname)[1][1:]
+    cmd = ['dot', '-T%s' % out_format, dot_fname]
+    exit_code = 42
+    with open(out_fname, "w") as out:
+        p = subprocess.Popen(
+            cmd, stdout = out, stderr = subprocess.PIPE )
+        nil, err = p.communicate()
+
+        sys.stderr.write(err)
+
+        exit_code = p.returncode
+        out.flush()
+    return bool(exit_code == 0)
+
+def svg2pdf(svg_fname, pdf_fname):
+    """Converts SVG to PDF with CairoSVG or ``convert(1)`` command.
+
+    Uses ``convert(1)`` from ImageMagick (https://www.imagemagick.org) for
+    conversion.  Returns ``True`` on success and ``False`` if an error occurred
+    (e.g. none of the conversion tool is available).
+
+    * ``svg_fname`` pathname of the input SVG file with extension (``.svg``)
+    * ``pdf_name``  pathname of the output PDF file with extension (``.pdf``)
+
+    """
+    cmd = [convert_cmd, svg_fname, pdf_fname]
+    p = subprocess.Popen(
+        cmd, stdout = out, stderr = subprocess.PIPE )
+    nil, err = p.communicate()
+
+    sys.stderr.write(err)
+
+    exit_code = p.returncode
+    return bool(exit_code == 0)
+
+
+# image handling
+# ---------------------
+
+def visit_kernel_image(self, node):    # pylint: disable=W0613
+    """Visitor of the ``kernel_image`` Node.
+
+    Handles the ``image`` child-node with the ``convert_image(...)``.
+    """
+    img_node = node[0]
+    convert_image(img_node, self)
+
+class kernel_image(nodes.General, nodes.Element):
+    """Node for ``kernel-image`` directive."""
+    pass
+
+class KernelImage(images.Image):
+    u"""KernelImage directive
+
+    Earns everything from ``.. image::`` directive, except *remote URI* and
+    *glob* pattern. The KernelImage wraps a image node into a
+    kernel_image node. See ``visit_kernel_image``.
+    """
+
+    def run(self):
+        env = self.state.document.settings.env
+
+        uri = self.arguments[0]
+        if uri.endswith('.*') or uri.find('://') != -1:
+            raise self.severe(
+                'Error in "%s: %s": glob pattern and remote images are not allowed'
+                % (self.name, uri))
+
+        # Tell sphinx of the dependency
+        env.note_dependency(os.path.abspath(uri))
+
+        result = images.Image.run(self)
+        if len(result) == 2 or isinstance(result[0], nodes.system_message):
+            return result
+        (image_node,) = result
+        # wrap image node into a kernel_image node / see visitors
+        node = kernel_image('', image_node)
+        return [node]
+
+# figure handling
+# ---------------------
+
+def visit_kernel_figure(self, node):   # pylint: disable=W0613
+    """Visitor of the ``kernel_figure`` Node.
+
+    Handles the ``image`` child-node with the ``convert_image(...)``.
+    """
+    img_node = node[0][0]
+    convert_image(img_node, self)
+
+class kernel_figure(nodes.General, nodes.Element):
+    """Node for ``kernel-figure`` directive."""
+
+class KernelFigure(patches.Figure):
+    u"""KernelImage directive
+
+    Earns everything from ``.. figure::`` directive, except *remote URI* and
+    *glob* pattern.  The KernelFigure wraps a figure node into a kernel_figure
+    node. See ``visit_kernel_figure``.
+    """
+
+    def run(self):
+        env = self.state.document.settings.env
+
+        uri = self.arguments[0]
+        if uri.endswith('.*') or uri.find('://') != -1:
+            raise self.severe(
+                'Error in "%s: %s":'
+                ' glob pattern and remote images are not allowed'
+                % (self.name, uri))
+
+        # Tell sphinx of the dependency
+        env.note_dependency(os.path.abspath(uri))
+
+        result = patches.Figure.run(self)
+        if len(result) == 2 or isinstance(result[0], nodes.system_message):
+            return result
+        (figure_node,) = result
+        # wrap figure node into a kernel_figure node / see visitors
+        node = kernel_figure('', figure_node)
+        return [node]
+
+
+# render handling
+# ---------------------
+
+def visit_kernel_render(self, node):
+    """Visitor of the ``kernel_render`` Node.
+
+    If rendering tools available, save the markup of the ``literal_block`` child
+    node into a file and replace the ``literal_block`` node with a new created
+    ``image`` node, pointing to the saved markup file. Afterwards, handle the
+    image child-node with the ``convert_image(...)``.
+    """
+
+    verbose = self.builder.app.verbose
+    warn    = self.builder.warn
+    srclang = node.get('srclang')
+
+    verbose('visit kernel-render node lang: "%s"' % (srclang))
+
+    tmp_ext = RENDER_MARKUP_EXT.get(srclang, None)
+    if tmp_ext is None:
+        warn('kernel-render: "%s" unknow / include raw.' % (srclang))
+        return
+
+    literal_block = node[0]
+    code = literal_block.astext()
+
+    if tmp_ext:
+        hashobj   = code.encode('utf-8') #  str(node.attributes)
+        fname     = '%s-%s' % (srclang, sha1(hashobj).hexdigest())
+        tmp_fname = path.join(
+            self.builder.outdir, self.builder.imagedir, fname + tmp_ext)
+
+        if not path.isfile(tmp_fname):
+            mkdir(path.dirname(tmp_fname))
+            with open(tmp_fname, "w") as out:
+                out.write(code)
+
+        image_node = nodes.image(node.rawsource, **node.attributes)
+        image_node['uri'] = tmp_fname
+
+        literal_block.replace_self(image_node)
+        convert_image(image_node, self)
+
+
+class kernel_render(nodes.General, nodes.Inline, nodes.Element):
+    """Node for ``kernel-render`` directive."""
+    pass
+
+class KernelRender(patches.Figure):
+    u"""KernelRender directive
+
+    Render content by external tool.  Has all the options known from the
+    *figure*  directive, plus option ``caption``.  If ``caption`` has a
+    value, a figure node with the *caption* is inserted. If not, a image node is
+    inserted.
+
+    The KernelRender directive wraps the text of the directive into a
+    literal_block node and wraps it into a kernel_render node. See
+    ``visit_kernel_render``.
+    """
+    has_content = True
+    required_arguments = 1
+    optional_arguments = 0
+    final_argument_whitespace = False
+
+    # earn options from 'figure'
+    option_spec = patches.Figure.option_spec.copy()
+    option_spec['caption'] = directives.unchanged
+
+    def run(self):
+        return [self.build_node()]
+
+    def build_node(self):
+
+        srclang = self.arguments[0].strip()
+        if srclang not in RENDER_MARKUP_EXT.keys():
+            return [self.state_machine.reporter.warning(
+                'Unknow source language "%s", use one of: %s.' % (
+                    srclang, ",".join(RENDER_MARKUP_EXT.keys())),
+                line=self.lineno)]
+
+        code = '\n'.join(self.content)
+        if not code.strip():
+            return [self.state_machine.reporter.warning(
+                'Ignoring "%s" directive without content.' % (
+                    self.name),
+                line=self.lineno)]
+
+        node = kernel_render()
+        node['alt'] = self.options.get('alt','')
+        node['srclang'] = srclang
+        literal_node = nodes.literal_block(code, code)
+        node += literal_node
+
+        caption = self.options.get('caption')
+        if caption:
+            # parse cation's content
+            parsed = nodes.Element()
+            self.state.nested_parse(
+                ViewList([caption], source=''), self.content_offset, parsed)
+            caption_node = nodes.caption(
+                parsed[0].rawsource, '', *parsed[0].children)
+            caption_node.source = parsed[0].source
+            caption_node.line = parsed[0].line
+
+            figure_node = nodes.figure('', node)
+            for k,v in self.options.items():
+                figure_node[k] = v
+            figure_node += caption_node
+
+            node = figure_node
+
+        return node
+
-- 
2.11.0



More information about the dri-devel mailing list