Generating Assertions

Pynguin is able to generate regression assertions within its generated test cases based on the values that it observed during exeuction.

It supports three different modes for generating assertions, which can be selected using --assertion-generation {SIMPLE, MUTATION_ANALYSIS, NONE} (NONE to disable assertion generation).

Pynguin assumes that the following objects are assertable:

  • Enum values

  • Simple data types (int, str, bytes, bool and None)

  • Builtin collections (tuple, list, dict and set) that consist only of assertable elements.

Additionally, Pynguin can generate assertions for:

  • float values, which use a fuzzy comparison, i.e., pytest.approx(...)

  • Collections that are not assertable, by asserting the value returned from len(...)

When dealing with non assertable objects, Pynguin will try to assert something on the object’s public attributes. If this is also not possible, Pynguin will fallback to asserting that an object is not None.

Simple

The simple approach observes objects seen during test case exeuction. This includes:

  • function/method/constructor return values

  • static fields on used modules

  • static fields on seen return types

The simple approach executes every test case twice and keeps the intersection of assertions for each statement between both executions. This filters out trivially flaky assertions, e.g., strings that include memory locations.

Mutation Analysis

The mutation analysis approach is an extension of the simple approach. It uses a fork of MutPy to generate mutants of the module under test and executes every generated test case on every generated mutant.

From the assertions generated by the simple approach, it only keeps those that were not observed in every mutated execution, i.e., only if there is a mutant that violates an assertion, the assertion is seen as relevant.

Additional Filtering

Both approaches try to filter out non-relevant and/or trivially-flaky assertions. They also filter out assertions that are stale, i.e., assertions that do not change during subsequent statement executions. For example, consider the following test case based on the quickstart Queue example:

int_0 = 42
queue_0 = module_0.Queue(int_0)
assert queue_0.max == 42
# ...
int_1 = 1337
queue_0.enqueue(int_1)
assert queue_0.max == 42
# ...
queue_0.enqueue(int_1)
assert queue_0.max == 42

Here, the max attribute of queue_0 has a constant value. However, Pynguin cannot know this, thus it adds an assertion on this value after each statement. Such an assertion can still be relevant, e.g., when we want to assert that a value remains constant, but they also clutter the test cases quite a bit. By default, both approaches remove such stale assertions. This can be changed using the option --allow-stale-assertions. This option is turned off by default, such that Pynguin only generates assertions for things that change between statement executions.