Few helper functions

It is quite common practice to store some data required for test in a repository as individual files. First of all it could be some input data for tests. Secondly, files to check a test output somehow.

I’ve found the following bunch of functions are pretty useful:

import pathlib

def data_dir():
    return pathlib.Path(__file__).parent / 'data'

def output_dir():
    return data_dir() / 'output'

def expected_results_dir():
    return data_dir() / 'expected-results'

def make_data_filename(filename):
    return data_dir() / filename

I put ‘em to test/conftest.py usually.

A simple way to assert test output

Since beginning of my experience with pytest I’ve looked for an easy way to match test output against an expected text or pattern. When I found nothing in docs and nothing interesting in StackOverflow and Google, I decided to write my own wheel ;-)

Add CLI option to store patterns

I like this featue in Boost Unit Testing Framework: --save_pattern CLI option, instead of checking the captured output result, just store it into a file, which is usually kept in a VCS as expected result.

Now lets implement the same for pytest! ;-) I found a sample, so it was the easiest part:

# Add `--save-patterns` CLI boolean option
def pytest_addoption(parser):
    parser.addoption(
        '--save-patterns'
      , action='store_true'
      , help='store matching patterns instead of checking them'
      )

Attention

This function (and everything else in this article) must be defined in conftest.py file of your tests. Then pytest can find your extensions and you could use them in your tests.

So far, so good… but by itself this option do nothing…

Write a matching fixture

In fact it would be a named argument fixture. What I want is a simple as possible way to match a test output with some expectations. Good to know that pytest can capture test output for you. Then (close to the end of your test) you can read it:

    out, err = capfd.readouterr()

and have a text strings. But in my experience:

  • test’s output could be really long (to assert out == 'expected string)
  • output may change from run to run (e.g. timestamps in logged strings)

To address the first issue it would be nice to store an expected output in a file under VCS control. For the second case, stored expectation could be a regular expression to be matched against the test’s output.

How it may looks like?

def my_test(capfd, expected_out):
    # ... do some testing ...

    out, err = capfd.readouterr()

    assert expected_out == out

Ok, lets see how expected_out would look like… First of all it is a fixture name – i.e. a function accepting request parameter provided by pytest. Using request it’s possible to get a module/class/function name which requested this fixture. Lets use ‘em to form a path to expectations file, which is in turn located somewhere in test/data/expected-results/ directory of project sources. The next part is to make an equal comparable object and return it as a fixture “result”. Equal operator later will be used in assert expression, where the second expression is the captured output (of string type).

Ok, here is a full code:

import pathlib
import pytest
import re
import warnings

class _content_check_or_store_pattern:

    def __init__(self, filename, store):
        self._filename = filename
        self._store = store


    def _store_pattern_handle_error(fn):
        def _inner(self, text):
            # Check if `--save-patterns` has given to CLI
            if self._store:
                # Make directory to store pattern file if it doesn't exist yet
                if not self._filename.parent.exists():
                    self._filename.parent.mkdir(parents=True)

                # Store!
                self._filename.write_text(text)
                return True

            # Ok, this is the "normal" check:
            # - make sure the pattern file exists
            if not self._filename.exists():
                warnings.warn('Pattern file not found `{}`'.format(self._filename), RuntimeWarning)
                return False

            # - call wrapped function to
            return fn(self, text)

        return _inner


    @_store_pattern_handle_error
    def __eq__(self, text):
        expected_text = self._filename.read_text().strip()
        return expected_text == text


    @_store_pattern_handle_error
    def match(self, text):
        content = ' '.join(self._filename.read_text().strip().splitlines())
        what = re.compile(content)
        transformed_text = ' '.join(text.splitlines())
        return bool(what.match(transformed_text))


def _make_expected_filename(request, ext):
    result = expected_results_dir()

    if request.cls is not None:
        result /= request.cls.__name__

    result /= request.function.__name__ + ext

    return result


@pytest.fixture
def expected_out(request):
    return _content_check_or_store_pattern(
        _make_expected_filename(request, '.out')
      , request.config.getoption('--save-patterns')
      )


@pytest.fixture
def expected_err(request):
    return _content_check_or_store_pattern(
        _make_expected_filename(request, '.err')
      , request.config.getoption('--save-patterns')
      )

Hence in a test function my_test, expected_out is an instance of _content_check_or_store_pattern, so one can use assert expected_out == out! For the fist time it is recommended to run py.test with --save-patters option and particular test name to execute – just to overwrite only one expectations file (and do not touch others):

$ pytest --save-patterns test/some_test.py::my_test

If test output has changed parts, then a better way is to use match() method:

def my_test(capfd, expected_out):
    # ... do some testing ...

    out, err = capfd.readouterr()

    assert expected_out.match(out)

Run pytest to store initial pattern, then edit the test/data/expected-results/my_test.out to replace mutable parts with .* regex’s wildcards (or some other, more precise expressions).

Some examples could be found in my tcct project ;-)



blog comments powered by Disqus