Running tests against multiple versions of a Python dependency in GitHub Actions

My datasette-export-notebook plugin worked fine in the stable release of Datasette, currently version 0.64.3, but failed in the Datasette 1.0 alphas. Here's the issue describing the problem.

Here's the pattern I figured out for running the tests in GitHub Actions against both Datasette versions. This is my full test.yml from that repository:

name: Test

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11"]
        datasette-version: ["<=1.0a0", ">=1.0a0"]
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
        cache: pip
        cache-dependency-path: setup.py
    - name: Install dependencies
      run: |
        pip install -e '.[test]'
        pip install 'datasette${{ matrix.datasette-version }}'
    - name: Run tests
      run: |
        pytest

The trick here is to set up a matrix for datasette-version (to accompany my existing python-version one) defining these two installation specifiers:

datasette-version: ["<=1.0a0", ">=1.0a0"]

Then later I use those to install the specified version of Datasette like this:

pip install 'datasette${{ matrix.datasette-version }}'

The single quotes there are important - without them my shell got confused by the <= and >= symbols.

The end result of this is that tests run against the highest Datasette release in the 0.x series, and also against the highest release in the 1.x series, including alphas if no 1.x stable release is out yet.

Adding extra version information to the pytest report

When using this pattern, it can be useful to include the Datasette version in the output of the pytest command.

Here's an easy way to do that: add the following to tests/conftest.py:

import datasette


def pytest_report_header():
    return "Datasette: {}".format(datasette.__version__)

Running pytest will now output the following:

============================ test session starts ============================
platform darwin -- Python 3.9.17, pytest-7.4.2, pluggy-1.3.0
Datasette: 1.0a6
rootdir: /Users/...

Created 2023-09-15T12:11:13-07:00, updated 2024-01-12T21:54:18-08:00 · History · Edit

Running tests against multiple versions of a Python dependency in GitHub Actions

Adding extra version information to the pytest report

Related