.. _assertions: Generating Assertions ===================== Pynguin is able to generate *regression* assertions within its generated test cases based on the values that it observed during exeuction. It supports three different modes for generating assertions, which can be selected using ``--assertion-generation {SIMPLE, MUTATION_ANALYSIS, NONE}`` (``NONE`` to disable assertion generation). Pynguin assumes that the following objects are assertable: * Enum values * Simple data types (``int``, ``str``, ``bytes``, ``bool`` and ``None``) * Builtin collections (``tuple``, ``list``, ``dict`` and ``set``) that consist only of assertable elements. Additionally, Pynguin can generate assertions for: * ``float`` values, which use a fuzzy comparison, i.e., ``pytest.approx(...)`` * Collections that are not assertable, by asserting the value returned from ``len(...)`` When dealing with non assertable objects, Pynguin will add an assertion on the type of the object. Additionally, it will try to assert something on the object's public attributes. Simple ------ The simple approach observes objects seen during test case exeuction. This includes: * function/method/constructor return values * static fields on used modules * static fields on seen return types The simple approach executes every test case once and creates assertions from the observerd states. Afterwards every test is executed again with the previously generated assertions. Only the assertions that hold in this execution are kept. This filters out trivially flaky assertions, e.g., strings that include memory locations. .. _mutation_analysis: Mutation Analysis ----------------- The mutation analysis approach is an extension of the simple approach. It uses a fork of `MutPy `_ to generate mutants of the module under test and executes every generated test case on every generated mutant. From the assertions generated by the simple approach, it only keeps those that failed on at least one mutant, i.e., only if there is a mutant that violates an assertion, the assertion is seen as relevant. Additional Filtering -------------------- Both approaches try to filter out non-relevant and/or trivially-flaky assertions. They also filter out assertions that are stale, i.e., assertions that do not change during subsequent statement executions. For example, consider the following test case based on the :ref:`quickstart ` ``Queue`` example: .. code-block:: python int_0 = 42 queue_0 = module_0.Queue(int_0) assert queue_0.max == 42 # ... int_1 = 1337 queue_0.enqueue(int_1) assert queue_0.max == 42 # ... queue_0.enqueue(int_1) assert queue_0.max == 42 Here, the ``max`` attribute of ``queue_0`` has a constant value. However, Pynguin cannot know this, thus it adds an assertion on this value after each statement. Such an assertion can still be relevant, e.g., when we want to assert that a value remains constant, but they also clutter the test cases quite a bit. By default, both approaches remove such stale assertions. This can be changed using the option ``--allow-stale-assertions``. This option is turned off by default, such that Pynguin only generates assertions for things that change between statement executions. Expected vs Unexpected Exceptions --------------------------------- Python, similar to many other programming languages, does not have a concept to declare expected or unexpected exceptions, as it is known from the Java world (called *checked* and *unchecked* there). Still the problem is highly relevant when generating assertions that check for an exception. We decided to define expected and unexpected exceptions in the context of Pynguin as follows (for simplicity, we talk about functions here, but this also includes methods inside classes among others): * An *expected exception* is an exception that is explicitly raised in a function but not caught. This can be done by the ``raise`` keyword without having a surrounding ``try``-``finally`` block. We furthermore call an exception expected if it is mentioned in the docstring of the function that the function raises such an exception. * An *unexpected exception* is an exception that is not explicitly raised in a function. This can happen, for example, because the exception is raised in another function that is called by the original function. We use the above definitions to generate specific assertions in case an exception is raised during test-case execution. For an expected exception of type ``ExampleException`` that is raised during execution of a function ``example``, Pynguin will generate the following statements in the resulting test case: .. code-block:: python with pytest.raises(ExampleException): example() For an unexpected exception that is raised during execution of a function ``example``, Pynguin will annotate the test function like follows: .. code-block:: python @pytest.mark.xfail(strict=True) def test_case_0(): example() Both variants use functionality from the `PyTest `_ framework: The ``pytest.raises`` function is used by PyTest to assert for an `expected exception `_; the ``pytest.mark.xfail`` decorator is used by PyTest to mark test functions as `expected to fail `_. For a user of Pynguin the latter is a clear hint to manually inspect the generated test case. If in the user's opinion the exception is something that is actually expected it is now the user's responsibility to change the code to a similar code that checks for an expected exception if they want to use the test case in their code base. One further type of expected assertion can be an ``AssertionError``. This exception type is expected if the source code of the tested function contains an ``assert`` statement.