Coverage measurement

Pynguin uses bytecode instrumentation to measure code coverage. For this, Pynguin modifies the bytecode of the module under test after it was compiled to bytecode but before it is executed.

Pynguin currently has support for branch and line coverage, which can be selected using --coverage-metrics, for example, one can use --coverage-metrics BRANCH LINE to target both metrics.

Coverage.py is the standard tool for measuring line and branch coverage in Python. Its measurements are based on executed lines (line coverage) and the transitions between them (branch coverage). Since Pynguin measures coverage at the bytecode level, there are some difference between its measured values and those of Coverage.py. In order to make it easier to see what parts have been (partially) covered by Pynguin, one can set the option --create-coverage-report True to generate a coverage report similar to the one of Coverage.py. The report is created at the location specified by --report-dir (./pynguin-report/ by default).

A report for the quickstart Queue example might look like this:

../_images/cov-report-queue.png

Where the marker colours have the following meaning:

  • Green: Everything located at that line has been covered

  • Orange: Not all branches located at that line have been covered, for example, line 26 did not jump to line 27.

  • Red: Nothing at that line has been covered.

Hovering over the line number of a coloured line displays a tooltip with detailed information, which can be seen in the following example:

../_images/cov-report-triangle.png

Pynguin measures branch coverage on the basis of code objects and their bytecode instructions. If the instructions of a code object contain no conditional jump, then Pynguin will count it as a branchless code object, which is seen as covered if it was executed at least once. For actual branches, Pynguin requires that each conditional jump is taken and not taken at least once.

Markers for branchless code objects are placed at the first line that belongs to the respective code object. A module itself is also a code object that is executed on import, for example, the module containing the triangle function is a branchless code object and thus has a green marker at line 1.

Note

Code objects are nested within each other, that means that the code object of the module contains the code objects of the functions defined directly within it in its constant table an so on.

For branches, the markers are placed at the line where the branching occurs, for example, the triangle function has four branches on line 10, due to a compound predicate (x == y and y == z) and short circuiting within the bytecode. Note that this is different to Coverage.py, which would only assign two branches to this line, that is, either a jump to line 11 or 12. Furthermore, Coverage.py recognizes only branches that cause a line transition, for example, the expression assert size_max > 0 contains a branch at the bytecode level, as it is equivalent to the following source code:

if size_max <= 0:
   raise AssertionError()

Such a branch which jumps within the same line is recognized by Pynguin but not by Coverage.py.

For line coverage the markers are placed at the respective lines. However, only lines that actually contain something that is compiled to bytecode instructions are seen as relevant, for example, the else: label in line 14 does not contain anything executable and is thus ignored.

By default, Pynguin will take into account the code annotations # pragma: no cover supported by Coverage.py and its own annotations # pynguin: no cover which only disable coverage for Pynguin. These annotations can be enabled/disabled using the arguments --enable-inline-pynguin-no-cover and --enable-inline-pragma-no-cover. Furthermore, Pynguin automatically excludes certain code blocks from its coverage goals that are known to be unreachable during test generation:

  • if __name__ == "__main__": blocks

  • if TYPE_CHECKING: blocks (including typing.TYPE_CHECKING and types.TYPE_CHECKING)

This is because Pynguin generates tests by importing the module under test, and code within such blocks is only executed when the module is run as a script or during static type checking. Excluding them allows Pynguin to reach 100% coverage and terminate the search early once all reachable code is covered in some cases. In other cases (e.g. when there are unreachable code), Pynguin will nevertheless continue the search until the time budget is exhausted.

Note that this automatic exclusion might result in Pynguin reporting a different coverage than other tools like pytest-cov (which uses Coverage.py) if those tools are not configured to exclude these blocks as well.

You can also disable some functions, methods or classes by specifying their qualified name in the --no-cover argument. It is also possible to do the opposite and cover only some functions, methods or classes with the --only-cover argument.