Quickstart

Eager to start? Make sure that Pynguin is installed properly.

Warning

Pynguin actually executes the code of the module under test. That means, if the code you want to generate tests for does something bad, for example wipes your disk, there is nothing that prevents it from doing so! This also includes code that is transitively imported by the module under test.

To mitigate this issue, we recommend running Pynguin in a Docker container with appropriate mounts from the host system’s file system. See the pynguin-docker.sh script in Pynguin’s source repository for documentation on the necessary mounts.

To help prevent harming the system that runs Pynguin, its CLI will immediately abort unless the environment variable PYNGUIN_DANGER_AWARE is set. In setting this variable, you acknowledge that you are aware of the possible dangers of executing code with random inputs. The assigned value can be arbitrary; Pynguin solely checks whether the variable is defined.

We do not provide any support and are not responsible if you break your computer by executing Pynguin on some random code from the internet! Be careful and check the code before actually executing it—which is good advice anyway.

Developers: If you know of a similar technique to Java’s security manager mechanism in Python, which we can use to mitigate this issue, please let us know.

A Simple Example

For a first impression, we use the bundled example file and generate tests for it. Note that this assumes that you have the source code checked out, installed Pynguin properly—as mentioned before, we recommend a virtual environment, which needs to be sourced manually—and that your shell is pointing to the root directory of Pynguin’s source repository. We run all commands on a command-line shell where we assume that the environment variable PYNGUIN_DANGER_AWARE is set.

Note

We don’t use docker in our examples, because we know that our examples do not contain or use code that might harm our system. But for unknown code we highly recommend using some form of isolation.

First, let’s look at the code of the example file (which is located in docs/source/_static/example.py):

1def triangle(x: int, y: int, z: int) -> str:
2    if x == y == z:
3        return "Equilateral triangle"
4    elif x == y or y == z or x == z:
5        return "Isosceles triangle"
6    else:
7        return "Scalene triangle"

The example is the classical triangle example from courses on Software Testing, which yields for three given integers—assumed to be the lengths of the triangle’s edges—what type of triangle it is. Note that we have annotated all parameter and return types, according to PEP 484.

Before we can start, we create a directory for the output (this assumes you are on a Linux or macOS machine, but similar can be done on Windows) using the command line:

$ mkdir -p /tmp/pynguin-results

We will now invoke Pynguin (using its default test-generation algorithm) to let it generate test cases (we use \ and the line breaks for better readability here, you can just omit them and type everything in one line):

$ pynguin \
    --project-path ./docs/source/_static \
    --output-path /tmp/pynguin-results \
    --module-name example

This runs for a moment without showing any output. Thus, to have some more verbose output we add the -v parameter:

$ pynguin \
    --project-path ./docs/source/_static \
    --output-path /tmp/pynguin-results \
    --module-name example \
    -v

The output on the command line might be something like the following:

[12:42:45] INFO     Start Pynguin Test Generation…                                                                                     generator.py:96
           INFO     Using seed 1636113559178642000                                                                                    generator.py:174
           INFO     Collecting constants from SUT.                                                                                    generator.py:181
           INFO     Using strategy: Algorithm.DYNAMOSA                                                               generationalgorithmfactory.py:235
           INFO     Instantiated 11 fitness functions                                                                generationalgorithmfactory.py:311
           INFO     Using CoverageArchive                                                                            generationalgorithmfactory.py:279
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                         generationalgorithmfactory.py:254
           INFO     Using stopping condition: StoppingCondition.MAX_TIME                                              generationalgorithmfactory.py:90
           INFO     Using crossover function: SinglePointRelativeCrossOver                                           generationalgorithmfactory.py:267
           INFO     Using ranking function: RankBasedPreferenceSorting                                               generationalgorithmfactory.py:287
           INFO     Start generating test cases                                                                                       generator.py:264
           INFO     Iteration:     0, Coverage: 1.000000                                                                          searchobserver.py:66
           INFO     Algorithm stopped before using all resources.                                                                     generator.py:269
           INFO     Stop generating test cases                                                                                        generator.py:270
           INFO     Start generating assertions                                                                                       generator.py:294
           INFO     Setup mutation controller                                                                                    mutationadapter.py:66
           INFO     Build AST for example                                                                                        mutationadapter.py:52
           INFO     Mutate module example                                                                                        mutationadapter.py:54
           INFO     Generated 14 mutants                                                                                         mutationadapter.py:62
           INFO     Running tests on mutant   1/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   2/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   3/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   4/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   5/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   6/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   7/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   8/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   9/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  10/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  11/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  12/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  13/14                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  14/14                                                                           assertiongenerator.py:158
           INFO     Export 5 successful test cases to /tmp/pynguin-results/test_example.py                                            generator.py:311
           INFO     Export 0 failing test cases to /tmp/pynguin-results/test_example_failing.py                                       generator.py:321
           INFO     Writing statistics                                                                                               statistics.py:350
           INFO     Stop Pynguin Test Generation…                                                                                      generator.py:99

The first few lines show that Pynguin starts, that it has not gotten any seed for its (pseudo) random-number generator, followed by the configuration options that are used for its DYNAMOSA algorithm. We can also see that it ran zero iterations of that algorithm, i.e., the initial random test cases were sufficient to cover all branches. This was to be expected, since the triangle example can be trivially covered with tests. Pynguin created assertions using Mutation Analysis. The output then concludes with its results: Five test cases were written to /tmp/pynguin/results/test_example.py, which look like the following (the result can differ on your machine):

 1# Automatically generated by Pynguin.
 2import example as module_0
 3
 4
 5def test_case_0():
 6    int_0 = 1484
 7    int_1 = 1905
 8    str_0 = module_0.triangle(int_1, int_1, int_1)
 9    assert str_0 == "Equilateral triangle"
10    int_2 = -3613
11    int_3 = 1408
12    str_1 = module_0.triangle(int_0, int_2, int_3)
13    assert str_1 == "Scalene triangle"
14
15
16def test_case_1():
17    int_0 = -3586
18    int_1 = -656
19    str_0 = module_0.triangle(int_0, int_1, int_0)
20    assert str_0 == "Isosceles triangle"
21
22
23def test_case_2():
24    int_0 = 65
25    int_1 = -1301
26    str_0 = module_0.triangle(int_0, int_0, int_1)
27    assert str_0 == "Isosceles triangle"
28    int_2 = -928
29    int_3 = 1261
30    str_1 = module_0.triangle(int_2, int_2, int_3)
31    assert str_1 == "Isosceles triangle"
32
33
34def test_case_3():
35    int_0 = 53
36    int_1 = 726
37    int_2 = -771
38    int_3 = -4443
39    str_0 = module_0.triangle(int_2, int_3, int_3)
40    assert str_0 == "Isosceles triangle"
41    str_1 = module_0.triangle(int_1, int_2, int_1)
42    assert str_1 == "Isosceles triangle"
43    int_4 = 1386
44    int_5 = None
45    str_2 = module_0.triangle(int_4, int_5, int_0)
46    assert str_2 == "Scalene triangle"
47    str_3 = module_0.triangle(int_0, int_0, int_1)
48    assert str_3 == "Isosceles triangle"
49    str_4 = module_0.triangle(int_1, int_1, int_0)
50    assert str_4 == "Isosceles triangle"
51
52
53def test_case_4():
54    int_0 = 443
55    int_1 = 1681
56    int_2 = 2773
57    str_0 = module_0.triangle(int_0, int_1, int_2)
58    assert str_0 == "Scalene triangle"

We can see that each test case consists of one or more invocations of the triangle function and that there are assertions that check for the correct return value.

Note

As of version 0.6.0, Pynguin is able to generate assertions for simple data types (int, float, str, bytes and bool), as well as checks for None return values.

Note

As of version 0.13.0, Pynguin also provides a better assertion generation based on mutation. This allows to generate assertions also for more complex data types, see assertions for more details.

A more complex example

The above triangle example is really simple and could also be covered by a simple fuzzing tool. Thus, we now look at a more complex example: An implementation of a Queue for int elements. (located in docs/source/_static/queue_example.py):

 1import array
 2from typing import Optional
 3
 4
 5class Queue:
 6    def __init__(self, size_max: int) -> None:
 7        assert size_max > 0
 8        self.max = size_max
 9        self.head = 0
10        self.tail = 0
11        self.size = 0
12        self.data = array.array("i", range(size_max))
13
14    def empty(self) -> bool:
15        return self.size != 0
16
17    def full(self) -> bool:
18        return self.size == self.max
19
20    def enqueue(self, x: int) -> bool:
21        if self.size == self.max:
22            return False
23        self.data[self.tail] = x
24        self.size += 1
25        self.tail += 1
26        if self.tail == self.max:
27            self.tail = 0
28        return True
29
30    def dequeue(self) -> Optional[int]:
31        if self.size == 0:
32            return None
33        x = self.data[self.head]
34        self.size -= 1
35        self.head += 1
36        if self.head == self.max:
37            self.head = 0
38        return x

Testing this queue is more complex. One needs to instantiate it, add items, etc. Similar to the triangle example, we start Pynguin with the following command:

$ pynguin \
    --project-path ./docs/source/_static/ \
    --output-path /tmp/pynguin-results \
    --module-name queue_example \
    -v \
    --seed 1629381673714481067

Note

We used a predefined seed here, because we know that Pynguin requires less iterations with this seed in this specific example and version. This was done to get a clearer log.

The command yields the following output:

[12:56:08] INFO     Start Pynguin Test Generation…                                                                                     generator.py:96
           INFO     Using seed 1629381673714481067                                                                                    generator.py:174
           INFO     Collecting constants from SUT.                                                                                    generator.py:181
           INFO     Using strategy: Algorithm.DYNAMOSA                                                               generationalgorithmfactory.py:235
           INFO     Instantiated 14 fitness functions                                                                generationalgorithmfactory.py:311
           INFO     Using CoverageArchive                                                                            generationalgorithmfactory.py:279
           INFO     Using selection function: Selection.TOURNAMENT_SELECTION                                         generationalgorithmfactory.py:254
           INFO     Using stopping condition: StoppingCondition.MAX_TIME                                              generationalgorithmfactory.py:90
           INFO     Using crossover function: SinglePointRelativeCrossOver                                           generationalgorithmfactory.py:267
           INFO     Using ranking function: RankBasedPreferenceSorting                                               generationalgorithmfactory.py:287
           INFO     Start generating test cases                                                                                       generator.py:264
           INFO     Iteration:     0, Coverage: 0.785714                                                                          searchobserver.py:66
           INFO     Iteration:     1, Coverage: 0.785714                                                                          searchobserver.py:72
           INFO     Iteration:     2, Coverage: 0.785714                                                                          searchobserver.py:72
           INFO     Iteration:     3, Coverage: 0.785714                                                                          searchobserver.py:72
           INFO     Iteration:     4, Coverage: 0.785714                                                                          searchobserver.py:72
[12:56:09] INFO     Iteration:     5, Coverage: 0.857143                                                                          searchobserver.py:72
           INFO     Iteration:     6, Coverage: 0.857143                                                                          searchobserver.py:72
           INFO     Iteration:     7, Coverage: 0.928571                                                                          searchobserver.py:72
           INFO     Iteration:     8, Coverage: 1.000000                                                                          searchobserver.py:72
           INFO     Algorithm stopped before using all resources.                                                                     generator.py:269
           INFO     Stop generating test cases                                                                                        generator.py:270
           INFO     Start generating assertions                                                                                       generator.py:294
           INFO     Setup mutation controller                                                                                    mutationadapter.py:66
           INFO     Build AST for queue_example                                                                                  mutationadapter.py:52
           INFO     Mutate module queue_example                                                                                  mutationadapter.py:54
           INFO     Generated 30 mutants                                                                                         mutationadapter.py:62
           INFO     Running tests on mutant   1/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   2/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   3/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   4/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   5/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   6/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   7/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   8/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant   9/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  10/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  11/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  12/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  13/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  14/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  15/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  16/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  17/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  18/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  19/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  20/30                                                                           assertiongenerator.py:158
[12:56:10] INFO     Running tests on mutant  21/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  22/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  23/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  24/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  25/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  26/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  27/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  28/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  29/30                                                                           assertiongenerator.py:158
           INFO     Running tests on mutant  30/30                                                                           assertiongenerator.py:158
           INFO     Export 3 successful test cases to /tmp/pynguin-results/test_queue_example.py                                      generator.py:311
           INFO     Export 4 failing test cases to /tmp/pynguin-results/test_queue_example_failing.py                                 generator.py:321
           INFO     Writing statistics                                                                                               statistics.py:350
           INFO     Stop Pynguin Test Generation…                                                                                      generator.py:99

We can see that the DYNAMOSA algorithm had to perform nine iterations to fully cover the Queue example with the given seed. We can also see that Pynguin generated three successful testcases:

 1# Automatically generated by Pynguin.
 2import queue_example as module_0
 3
 4
 5def test_case_0():
 6    int_0 = 1845
 7    queue_0 = module_0.Queue(int_0)
 8    assert queue_0.max == 1845
 9    assert queue_0.head == 0
10    assert queue_0.tail == 0
11    assert queue_0.size == 0
12    assert len(queue_0.data) == 1845
13    bool_0 = queue_0.empty()
14    assert bool_0 is False
15    int_1 = 484
16    queue_1 = module_0.Queue(int_1)
17    assert queue_1.max == 484
18    assert queue_1.head == 0
19    assert queue_1.tail == 0
20    assert queue_1.size == 0
21    assert len(queue_1.data) == 484
22
23
24def test_case_1():
25    int_0 = 2311
26    queue_0 = module_0.Queue(int_0)
27    assert queue_0.max == 2311
28    assert queue_0.head == 0
29    assert queue_0.tail == 0
30    assert queue_0.size == 0
31    assert len(queue_0.data) == 2311
32    bool_0 = queue_0.full()
33    assert bool_0 is False
34    queue_1 = module_0.Queue(int_0)
35    assert queue_1.max == 2311
36    assert queue_1.head == 0
37    assert queue_1.tail == 0
38    assert queue_1.size == 0
39    assert len(queue_1.data) == 2311
40    queue_2 = module_0.Queue(int_0)
41    assert queue_2.max == 2311
42    assert queue_2.head == 0
43    assert queue_2.tail == 0
44    assert queue_2.size == 0
45    assert len(queue_2.data) == 2311
46    bool_1 = queue_1.full()
47    assert bool_1 is False
48    int_1 = 2477
49    bool_2 = queue_1.enqueue(int_1)
50    assert bool_2 is True
51    assert queue_1.tail == 1
52    assert queue_1.size == 1
53    bool_3 = queue_1.empty()
54    assert bool_3 is True
55    optional_0 = queue_2.dequeue()
56    assert optional_0 is None
57
58
59def test_case_2():
60    int_0 = 2311
61    queue_0 = module_0.Queue(int_0)
62    assert queue_0.max == 2311
63    assert queue_0.head == 0
64    assert queue_0.tail == 0
65    assert queue_0.size == 0
66    assert len(queue_0.data) == 2311
67    bool_0 = queue_0.full()
68    assert bool_0 is False
69    queue_1 = module_0.Queue(int_0)
70    assert queue_1.max == 2311
71    assert queue_1.head == 0
72    assert queue_1.tail == 0
73    assert queue_1.size == 0
74    assert len(queue_1.data) == 2311
75    queue_2 = module_0.Queue(int_0)
76    assert queue_2.max == 2311
77    assert queue_2.head == 0
78    assert queue_2.tail == 0
79    assert queue_2.size == 0
80    assert len(queue_2.data) == 2311
81    bool_1 = queue_1.full()
82    assert bool_1 is False
83    int_1 = 2477
84    bool_2 = queue_1.enqueue(int_1)
85    assert bool_2 is True
86    assert queue_1.tail == 1
87    assert queue_1.size == 1
88    bool_3 = queue_1.empty()
89    assert bool_3 is True
90    optional_0 = queue_1.dequeue()
91    assert optional_0 == 2477
92    assert queue_1.head == 1
93    assert queue_1.size == 0

And that it also generated four failing test cases, one of which looks this:

 1def test_case_1():
 2    try:
 3        int_0 = 3390
 4        queue_0 = module_0.Queue(int_0)
 5        assert queue_0.max == 3390
 6        assert queue_0.head == 0
 7        assert queue_0.tail == 0
 8        assert queue_0.size == 0
 9        assert len(queue_0.data) == 3390
10        bool_0 = queue_0.full()
11        assert bool_0 is False
12        int_1 = -475
13        queue_1 = module_0.Queue(int_1)
14    except BaseException:
15        pass

Failing test cases hereby are test cases that raised an exception during their execution. For now, Pynguin cannot know if an exception is expected program behavior, caused by an invalid input or an actual fault. Thus, these test cases are wrapped in try-except blocks and should be manually inspected.

Note

Generated test cases may contain a lot of superfluous statements. Future versions of Pynguin will try minimize test cases as much as possible while retaining their coverage.

Also many generated assertions might be redundant. Minimising these is open for a future release of Pynguin, too.