Generating Assertions

Pynguin is able to generate regression assertions within its generated test cases based on the values that it observed during exeuction.

It supports three different modes for generating assertions, which can be selected using --assertion-generation {SIMPLE, MUTATION_ANALYSIS, NONE} (NONE to disable assertion generation).

Pynguin assumes that the following objects are assertable:

  • Enum values

  • Simple data types (int, str, bytes, bool and None)

  • Builtin collections (tuple, list, dict and set) that consist only of assertable elements.

Additionally, Pynguin can generate assertions for:

  • float values, which use a fuzzy comparison, i.e., pytest.approx(...)

  • Collections that are not assertable, by asserting the value returned from len(...)

When dealing with non assertable objects, Pynguin will add an assertion on the type of the object. Additionally, it will try to assert something on the object’s public attributes.

Simple

The simple approach observes objects seen during test case exeuction. This includes:

  • function/method/constructor return values

  • static fields on used modules

  • static fields on seen return types

The simple approach executes every test case once and creates assertions from the observerd states. Afterwards every test is executed again with the previously generated assertions. Only the assertions that hold in this execution are kept. This filters out trivially flaky assertions, e.g., strings that include memory locations.

Mutation Analysis

The mutation analysis approach is an extension of the simple approach. It uses a fork of MutPy to generate mutants of the module under test and executes every generated test case on every generated mutant.

From the assertions generated by the simple approach, it only keeps those that failed on at least one mutant, i.e., only if there is a mutant that violates an assertion, the assertion is seen as relevant.

Additional Filtering

Both approaches try to filter out non-relevant and/or trivially-flaky assertions. They also filter out assertions that are stale, i.e., assertions that do not change during subsequent statement executions. For example, consider the following test case based on the quickstart Queue example:

int_0 = 42
queue_0 = module_0.Queue(int_0)
assert queue_0.max == 42
# ...
int_1 = 1337
queue_0.enqueue(int_1)
assert queue_0.max == 42
# ...
queue_0.enqueue(int_1)
assert queue_0.max == 42

Here, the max attribute of queue_0 has a constant value. However, Pynguin cannot know this, thus it adds an assertion on this value after each statement. Such an assertion can still be relevant, e.g., when we want to assert that a value remains constant, but they also clutter the test cases quite a bit. By default, both approaches remove such stale assertions. This can be changed using the option --allow-stale-assertions. This option is turned off by default, such that Pynguin only generates assertions for things that change between statement executions.

Expected vs Unexpected Exceptions

Python, similar to many other programming languages, does not have a concept to declare expected or unexpected exceptions, as it is known from the Java world (called checked and unchecked there). Still the problem is highly relevant when generating assertions that check for an exception.

We decided to define expected and unexpected exceptions in the context of Pynguin as follows (for simplicity, we talk about functions here, but this also includes methods inside classes among others):

  • An expected exception is an exception that is explicitly raised in a function but not caught. This can be done by the raise keyword without having a surrounding try-finally block.

    We furthermore call an exception expected if it is mentioned in the docstring of the function that the function raises such an exception.

  • An unexpected exception is an exception that is not explicitly raised in a function. This can happen, for example, because the exception is raised in another function that is called by the original function.

We use the above definitions to generate specific assertions in case an exception is raised during test-case execution. For an expected exception of type ExampleException that is raised during execution of a function example, Pynguin will generate the following statements in the resulting test case:

with pytest.raises(ExampleException):
    example()

For an unexpected exception that is raised during execution of a function example, Pynguin will annotate the test function like follows:

@pytest.mark.xfail(strict=True)
def test_case_0():
    example()

Both variants use functionality from the PyTest framework: The pytest.raises function is used by PyTest to assert for an expected exception; the pytest.mark.xfail decorator is used by PyTest to mark test functions as expected to fail.

For a user of Pynguin the latter is a clear hint to manually inspect the generated test case. If in the user’s opinion the exception is something that is actually expected it is now the user’s responsibility to change the code to a similar code that checks for an expected exception if they want to use the test case in their code base.

One further type of expected assertion can be an AssertionError. This exception type is expected if the source code of the tested function contains an assert statement.