Remarks for Kernel developer

Starting with Linux Kernel v4.8 a sphinx-doc build is available to build formats like HTML from reStructuredText (reST) markup. The Makefile target htmldocs builds the HTML documentation:

make htmldocs

The sphinx extensions for this build, which are shipped by the kernel source tree, are placed in the Documentation/sphinx folder. Some of the LinuxDoc features are already a part of the Kernel’s Documentation/sphinx folder others not (yet). E.g. the sphinx-doc extensions flat-table, cdomain, kfigure and kernel-include are merged into Kernel’s source tree. On the other side, e.g. for parsing kernel-doc comments, the Linux Kernel build process uses a Perl script while LinuxDoc brings a python module with a kernel-doc parser.

One drawback of the Perl script is, that it fits not very well into sphinx-extensions which are normally written in python. As a stand-alone parser it might be good enough, but mostly the parser is driven by the kernel-doc directive and this can’t be done in-process when you need a separate process for the Perl interpreter. The only thing that such a parser can do, is to pipe its results to stdout, which can be read by the kernel-doc directive. For a starting point at that time (2016), the reuse of the given Perl parser in Kernel’s sources was good enough but nowerdays it has its limits. This was the time, I started a spin-off with a POC, implementing a python variant of the parser together with it’s kernel-doc directive.

There was also a attempt in 2017 where I send a RFC replacing the Perl parser with the python variant, some parts of the discussion are worth to mention:

Everything has evolved in the meantime. Meanwhile the LinuxDoc project is also used in other projects and covers more use-cases than only to build Kernel’s documentation. Anyway both implementations, the python and the Perl one support kernel-doc markup while the Python one always serves a superset over the Perl script from Kernel’s source. That’s why it is possible to (drop-in) replace Kernel’s sphinx-extensions and the Perl parser with the LinuxDoc extension. To see how, take a look at chapter Patch Linux Kernel Documentation.

It is worth to mention, that both scripts will produce different reST from the same kernel-doc markup. This is only logical and consistent, because the python variant is further more developed and integrates better into Sphinx.

To give you a picture of such a difference, lets take the example from introduction, take a look at output from the Perl variant below and compare it with the output of the Python variant.

.. c:function:: int foobar (int arg1, int arg2)

   short function description of foobar


``int arg1``
  Describe the first argument to foobar.

``int arg2``
  Describe the second argument to foobar.  One can provide multiple line
  descriptions for arguments.


A longer description, with more discussion of the function :c:func:`foobar()` that
might be useful to those using or modifying it.  Begins with empty comment
line and may include additional embedded empty comment lines.  Within, you
can refer other definitions (e.g. :c:type:`struct my_struct <my_struct>`,
:c:type:`typedef my_typedef <my_typedef>`,``CONSTANT``, $ENVVAR etc.).

The longer description can have multiple paragraphs and you can use reST
inline markups like *emphasise* and **emphasis strong**.  You also have reST
block markups like lists or literal available:

Ordered List:
- item one
- item two
- literal block::

     a + b --> x


Describe the return value of foobar.

As you can see, Perl’s output is a flatten stream of markups and emphasis while the the Python variant produce a structured and valid reST document. One might think, that we can fix the Perl script to be more structural. As far as I know, it is not possible to embed structured reST into the doctree without being in-process where you have access to the in memory doctree. I don’t know where the Kernel development goes in the future, IMO there are three choices:

  1. Stay with Perl implementation of the kernel-doc parser and fumble around with its limitations.

  2. Take over the Python implementation of the parser and it’s kernel-doc directive.

  3. Or simply add LinuxDoc as one external requirement more to the build chain and drop/ignore the extension currently in the source tree (for a POC see patch).