Writing tests with Copilot

I needed to write a relatively repetitive collection of tests, for a number of different possible error states.

The code I was testing looks like this:

columns = data.get("columns")
rows = data.get("rows")
row = data.get("row")
if not columns and not rows and not row:
    return _error(["columns, rows or row is required"])

if rows and row:
    return _error(["Cannot specify both rows and row"])

if columns:
    if rows or row:
        return _error(["Cannot specify columns with rows or row"])
    if not isinstance(columns, list):
        return _error(["columns must be a list"])
    for column in columns:
        if not isinstance(column, dict):
            return _error(["columns must be a list of objects"])
        if not column.get("name") or not isinstance(column.get("name"), str):
            return _error(["Column name is required"])
        if not column.get("type"):
            column["type"] = "text"
        if column["type"] not in self._supported_column_types:
            return _error(
                ["Unsupported column type: {}".format(column["type"])]
            )
    # No duplicate column names
    dupes = {c["name"] for c in columns if columns.count(c) > 1}
    if dupes:
        return _error(["Duplicate column name: {}".format(", ".join(dupes))])

I wanted to write tests for each of the error cases. I'd already constructed the start of a parameterized pytest test for these.

I got Copilot/GPT-3 to write most of the tests for me.

First I used VS Code to select all of the _error(...) lines. I pasted those into a new document and turned them into a sequence of comments, like this:

# Error: columns must be a list
# Error: columns must be a list of objects
# Error: Column name is required
# Error: Unsupported column type
# Error: Duplicate column name
# Error: rows must be a list
# Error: rows must be a list of objects
# Error: pk must be a string

I pasted those comments into my test file inside the existing list of parameterized tests, then wrote each test by adding a newline beneath a comment and hitting tab until Copilot had written the test for me.

It correctly guessed both the error assertion and the desired invalid input for each one!

Here's an animated screenshot:

The finished tests are here.

Created 2022-11-14T21:52:33-08:00, updated 2022-11-14T22:03:58-08:00 · History · Edit

Writing tests with Copilot

Related