Adding Sphinx autodoc to a project, and configuring Read The Docs to build it

I built a new API reference page today for sqlite-utils, using the Sphinx autodoc extension to extract docstrings from the code and use them to build a full class reference.

I've avoided this kind of documentation in the past because I think narrative prose is a much better way of providing documentation - but sqlite-utils already has detailed narrative prose, so I felt that adding reference documentation powered by docstrings could enhance that project - while also providing better inline document for tools such as Visual Studio Code and Jupyter.

Setting up autodoc

Getting autodoc working in my local documentation environment took a few steps.

First, I added "sphinx.ext.autodoc" to the extensions = [] list in my docs/conf.py file.

Then I created a new page in docs/reference.rst and added the following:

===============
 API Reference
===============

.. _reference_db_database:

sqlite_utils.db.Database
========================

.. autoclass:: sqlite_utils.db.Database
    :members:
    :undoc-members:

This was enough to start generating documentation for the sqlite_utils.db.Database class.

:members: means "show documentation for everything in the class that has a docstring".

:undoc-members: means "also include class members that don't have a docstring yet".

I added my own rST headings and reference links - this ensured that the classes would automatically show up in the table of contents for my documentation.

Live updates while editing docstrings

I use sphinx-autobuild to automatically rebuild and reload the browser while I'm editing documentation.

This has a known limitation that it won't pick up on edits made to the *.py files that should cause a reload.

I found a workaround: add -a to disable incremental builds (rebuilding everything) and --watch ../sqlite_utils to trigger a new build when any file in that folder was updated.

I added this to the Makefile is the docs/ directory:

livehtml:
    sphinx-autobuild -a -b html "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(0) --watch ../sqlite_utils

Then running make livehtml started a web server for the documentation that would refresh the browser any time I edited a docstring.

More autoclass options

I ended up using the following:

.. autoclass:: sqlite_utils.db.Database
    :members:
    :undoc-members:
    :show-inheritance:
    :special-members: __getitem__
    :exclude-members: use_counts_table, execute_returning_dicts, resolve_foreign_keys

:show-inheritance: adds a note about the inheritance hierarchy for the class.

By default __special__ methods like __getitem__ are not included. :special-members: fixes this for specified methods.

:exclude-members; accepts a comma-separated list of methods that shouldn't be shown in the documentation. I used this to remove some internal methods.

I could have renamed these to _use_counts_table but that would be a breaking API change, even though these methods are deliberately undocumented and shouldn't be used by anyone.

Source, not alphabetical order

The class methods default to being shown in alphabetical order - adding autodoc_member_order = "bysource" to docs/conf.py changed that to source order instead.

reStructuredText in the docstrings

Adding reStructuredText to the docstrings works for formatting the generated API documentation. You can also use :ref:reference-links. Here's an example of a docstring with more complex formatting:

    def rows_where(
        self,
        where: str = None,
        where_args: Optional[Union[Iterable, dict]] = None,
        order_by: str = None,
        select: str = "*",
        limit: int = None,
        offset: int = None,
    ) -> Generator[dict, None, None]:
        """
        Iterate over every row in this table or view that matches the specified where clause.
        Returns each row as a dictionary. See :ref:`python_api_rows` for more details.

        :param where: SQL where fragment to use, for example ``id > ?``
        :param where_args: Parameters to use with that fragment - an iterable for ``id > ?``
          parameters, or a dictionary for ``id > :id``
        :param order_by: Column or fragment of SQL to order by
        :param select: Comma-separated list of columns to select - defaults to ``*``
        :param limit: Integer number of rows to limit to
        :param offset: Integer for SQL offset
        """
        if not self.exists():
            # ...

If you add autodoc_typehints = "description" to the conf.py configuration the :param name: Description lines will be shown in a list of parameters that also includes the type hints:

I learned about this in sqlite-utils/issues/413 while trying to improve the display of complex type annotations.

Getting it working on Read The Docs

Read The Docs has the option to deploy branches - you can set that up in the "Versions" tab. I was working in an autodoc branch - I set that branch to be "Active" and "Hidden", so it would be built by Read The Docs but not linked to from anywhere.

My first attempt failed to render the generated documentation, because Read The Docs did not know where to import my code from.

I solved that by adding a .readthedocs.yaml configuration file to my repository containing the following:

version: 2

sphinx:
  configuration: docs/conf.py

python:
  version: "3.8"
  install:
  - method: pip
    path: .
    extra_requirements:
    - docs

The key thing here is that install block, which tells Read The Docs to run the equivalent of this before building the documentation:

pip install -e '.[docs]'

For my repository that installs the docs optional dependencies from the setup.py, but also installs the sqlite_utils Python package itself. That way Sphinx can find and import sqlite_utils.db.Database in order to generate the documentation.

The end result

The new page of documentation is now live at en/latest/reference.html. The pull request in which I figured this all out is sqlite-utils/pull/312.

Created 2021-08-10T16:31:15-07:00, updated 2022-03-11T10:06:59-08:00 · History · Edit