Quickstart¶
Eager to start? Make sure that Pynguin is installed properly.
Warning
Pynguin actually executes the code of the module under test. That means, if the code you want to generate tests for does something bad, for example wipes your disk, there is nothing that prevents it from doing so! This also includes code that is transitively imported by the module under test.
To mitigate this issue, we recommend running Pynguin in a Docker container with
appropriate mounts from the host system’s file system.
See the pynguin-docker.sh script in Pynguin’s source repository for documentation
on the necessary mounts.
To help prevent harming the system that runs Pynguin, its CLI will immediately abort
unless the environment variable PYNGUIN_DANGER_AWARE is set. In setting this
variable, you acknowledge that you are aware of the possible dangers of executing code
with random inputs. The assigned value can be arbitrary; Pynguin solely checks
whether the variable is defined.
We do not provide any support and are not responsible if you break your computer by executing Pynguin on some random code from the internet! Be careful and check the code before actually executing it—which is good advice anyway.
Developers: If you know of a similar technique to Java’s security manager mechanism in Python, which we can use to mitigate this issue, please let us know.
A Simple Example¶
For a first impression, we use the bundled example file and generate tests for it.
Note that this assumes that you have the source code checked out, installed Pynguin
properly—as mentioned before, we recommend a virtual environment, which needs to be
sourced manually—and that your shell is pointing to the root directory of Pynguin’s
source repository.
We run all commands on a command-line shell where we assume that the environment variable
PYNGUIN_DANGER_AWARE is set.
Note
We don’t use docker in our examples, because we know that our examples do not contain or use code that might harm our system. But for unknown code we highly recommend using some form of isolation.
First, let’s look at the code of the example file (which is located in
docs/source/_static/example.py):
1def triangle(x: int, y: int, z: int) -> str:
2 if x == y == z:
3 return "Equilateral triangle"
4 elif x == y or y == z or x == z:
5 return "Isosceles triangle"
6 else:
7 return "Scalene triangle"
The example is the classical triangle example from courses on Software Testing,
which yields for three given integers—assumed to be the lengths of the triangle’s
edges—what type of triangle it is.
Note that we have annotated all parameter and return types, according to
PEP 484.
Before we can start, we create a directory for the output (this assumes you are on a Linux or macOS machine, but similar can be done on Windows) using the command line:
$ mkdir -p /tmp/pynguin-results
We will now invoke Pynguin (using its default test-generation algorithm) to let
it generate test cases (we use \ and the line breaks for better readability here,
you can just omit them and type everything in one line):
$ pynguin \
--project-path ./docs/source/_static \
--output-path /tmp/pynguin-results \
--module-name example
This runs for a moment without showing any output. Thus, to have some more verbose
output we add the -v parameter:
$ pynguin \
--project-path ./docs/source/_static \
--output-path /tmp/pynguin-results \
--module-name example \
-v
The output on the command line might be something like the following:
[12:42:45] INFO Start Pynguin Test Generation… generator.py:96
INFO Using seed 1636113559178642000 generator.py:174
INFO Collecting constants from SUT. generator.py:181
INFO Using strategy: Algorithm.DYNAMOSA generationalgorithmfactory.py:235
INFO Instantiated 11 fitness functions generationalgorithmfactory.py:311
INFO Using CoverageArchive generationalgorithmfactory.py:279
INFO Using selection function: Selection.TOURNAMENT_SELECTION generationalgorithmfactory.py:254
INFO Using stopping condition: StoppingCondition.MAX_TIME generationalgorithmfactory.py:90
INFO Using crossover function: SinglePointRelativeCrossOver generationalgorithmfactory.py:267
INFO Using ranking function: RankBasedPreferenceSorting generationalgorithmfactory.py:287
INFO Start generating test cases generator.py:264
INFO Iteration: 0, Coverage: 1.000000 searchobserver.py:66
INFO Algorithm stopped before using all resources. generator.py:269
INFO Stop generating test cases generator.py:270
INFO Start generating assertions generator.py:294
INFO Setup mutation controller mutationadapter.py:66
INFO Build AST for example mutationadapter.py:52
INFO Mutate module example mutationadapter.py:54
INFO Generated 14 mutants mutationadapter.py:62
INFO Running tests on mutant 1/14 assertiongenerator.py:158
INFO Running tests on mutant 2/14 assertiongenerator.py:158
INFO Running tests on mutant 3/14 assertiongenerator.py:158
INFO Running tests on mutant 4/14 assertiongenerator.py:158
INFO Running tests on mutant 5/14 assertiongenerator.py:158
INFO Running tests on mutant 6/14 assertiongenerator.py:158
INFO Running tests on mutant 7/14 assertiongenerator.py:158
INFO Running tests on mutant 8/14 assertiongenerator.py:158
INFO Running tests on mutant 9/14 assertiongenerator.py:158
INFO Running tests on mutant 10/14 assertiongenerator.py:158
INFO Running tests on mutant 11/14 assertiongenerator.py:158
INFO Running tests on mutant 12/14 assertiongenerator.py:158
INFO Running tests on mutant 13/14 assertiongenerator.py:158
INFO Running tests on mutant 14/14 assertiongenerator.py:158
INFO Export 5 successful test cases to /tmp/pynguin-results/test_example.py generator.py:311
INFO Export 0 failing test cases to /tmp/pynguin-results/test_example_failing.py generator.py:321
INFO Writing statistics statistics.py:350
INFO Stop Pynguin Test Generation… generator.py:99
The first few lines show that Pynguin starts, that it has not gotten any seed for its
(pseudo) random-number generator, followed by the configuration
options that are used for its DYNAMOSA algorithm.
We can also see that it ran zero iterations of that algorithm, i.e.,
the initial random test cases were sufficient to cover all branches.
This was to be expected, since the triangle example can be trivially covered with tests.
Pynguin created assertions using Mutation Analysis.
The output then concludes with its results:
Five test cases were written to /tmp/pynguin/results/test_example.py, which look
like the following (the result can differ on your machine):
1# Automatically generated by Pynguin.
2import example as module_0
3
4
5def test_case_0():
6 int_0 = 1484
7 int_1 = 1905
8 str_0 = module_0.triangle(int_1, int_1, int_1)
9 assert str_0 == "Equilateral triangle"
10 int_2 = -3613
11 int_3 = 1408
12 str_1 = module_0.triangle(int_0, int_2, int_3)
13 assert str_1 == "Scalene triangle"
14
15
16def test_case_1():
17 int_0 = -3586
18 int_1 = -656
19 str_0 = module_0.triangle(int_0, int_1, int_0)
20 assert str_0 == "Isosceles triangle"
21
22
23def test_case_2():
24 int_0 = 65
25 int_1 = -1301
26 str_0 = module_0.triangle(int_0, int_0, int_1)
27 assert str_0 == "Isosceles triangle"
28 int_2 = -928
29 int_3 = 1261
30 str_1 = module_0.triangle(int_2, int_2, int_3)
31 assert str_1 == "Isosceles triangle"
32
33
34def test_case_3():
35 int_0 = 53
36 int_1 = 726
37 int_2 = -771
38 int_3 = -4443
39 str_0 = module_0.triangle(int_2, int_3, int_3)
40 assert str_0 == "Isosceles triangle"
41 str_1 = module_0.triangle(int_1, int_2, int_1)
42 assert str_1 == "Isosceles triangle"
43 int_4 = 1386
44 int_5 = None
45 str_2 = module_0.triangle(int_4, int_5, int_0)
46 assert str_2 == "Scalene triangle"
47 str_3 = module_0.triangle(int_0, int_0, int_1)
48 assert str_3 == "Isosceles triangle"
49 str_4 = module_0.triangle(int_1, int_1, int_0)
50 assert str_4 == "Isosceles triangle"
51
52
53def test_case_4():
54 int_0 = 443
55 int_1 = 1681
56 int_2 = 2773
57 str_0 = module_0.triangle(int_0, int_1, int_2)
58 assert str_0 == "Scalene triangle"
We can see that each test case consists of one or more invocations of the triangle function
and that there are assertions that check for the correct return value.
Note
As of version 0.6.0, Pynguin is able to generate assertions for simple data
types (int, float, str, bytes and bool), as well as checks for None
return values.
Note
As of version 0.13.0, Pynguin also provides a better assertion generation based on mutation. This allows to generate assertions also for more complex data types, see assertions for more details.
A more complex example¶
The above triangle example is really simple and could also be covered by a simple fuzzing tool.
Thus, we now look at a more complex example: An implementation of a Queue for int elements.
(located in docs/source/_static/queue_example.py):
1import array
2from typing import Optional
3
4
5class Queue:
6 def __init__(self, size_max: int) -> None:
7 assert size_max > 0
8 self.max = size_max
9 self.head = 0
10 self.tail = 0
11 self.size = 0
12 self.data = array.array("i", range(size_max))
13
14 def empty(self) -> bool:
15 return self.size != 0
16
17 def full(self) -> bool:
18 return self.size == self.max
19
20 def enqueue(self, x: int) -> bool:
21 if self.size == self.max:
22 return False
23 self.data[self.tail] = x
24 self.size += 1
25 self.tail += 1
26 if self.tail == self.max:
27 self.tail = 0
28 return True
29
30 def dequeue(self) -> int | None:
31 if self.size == 0:
32 return None
33 x = self.data[self.head]
34 self.size -= 1
35 self.head += 1
36 if self.head == self.max:
37 self.head = 0
38 return x
Testing this queue is more complex. One needs to instantiate it, add items, etc.
Similar to the triangle example, we start Pynguin with the following command:
$ pynguin \
--project-path ./docs/source/_static/ \
--output-path /tmp/pynguin-results \
--module-name queue_example \
-v \
--seed 1629381673714481067
Note
We used a predefined seed here, because we know that Pynguin requires less iterations with this seed in this specific example and version. This was done to get a clearer log.
The command yields the following output:
[12:56:08] INFO Start Pynguin Test Generation… generator.py:96
INFO Using seed 1629381673714481067 generator.py:174
INFO Collecting constants from SUT. generator.py:181
INFO Using strategy: Algorithm.DYNAMOSA generationalgorithmfactory.py:235
INFO Instantiated 14 fitness functions generationalgorithmfactory.py:311
INFO Using CoverageArchive generationalgorithmfactory.py:279
INFO Using selection function: Selection.TOURNAMENT_SELECTION generationalgorithmfactory.py:254
INFO Using stopping condition: StoppingCondition.MAX_TIME generationalgorithmfactory.py:90
INFO Using crossover function: SinglePointRelativeCrossOver generationalgorithmfactory.py:267
INFO Using ranking function: RankBasedPreferenceSorting generationalgorithmfactory.py:287
INFO Start generating test cases generator.py:264
INFO Iteration: 0, Coverage: 0.785714 searchobserver.py:66
INFO Iteration: 1, Coverage: 0.785714 searchobserver.py:72
INFO Iteration: 2, Coverage: 0.785714 searchobserver.py:72
INFO Iteration: 3, Coverage: 0.785714 searchobserver.py:72
INFO Iteration: 4, Coverage: 0.785714 searchobserver.py:72
[12:56:09] INFO Iteration: 5, Coverage: 0.857143 searchobserver.py:72
INFO Iteration: 6, Coverage: 0.857143 searchobserver.py:72
INFO Iteration: 7, Coverage: 0.928571 searchobserver.py:72
INFO Iteration: 8, Coverage: 1.000000 searchobserver.py:72
INFO Algorithm stopped before using all resources. generator.py:269
INFO Stop generating test cases generator.py:270
INFO Start generating assertions generator.py:294
INFO Setup mutation controller mutationadapter.py:66
INFO Build AST for queue_example mutationadapter.py:52
INFO Mutate module queue_example mutationadapter.py:54
INFO Generated 30 mutants mutationadapter.py:62
INFO Running tests on mutant 1/30 assertiongenerator.py:158
INFO Running tests on mutant 2/30 assertiongenerator.py:158
INFO Running tests on mutant 3/30 assertiongenerator.py:158
INFO Running tests on mutant 4/30 assertiongenerator.py:158
INFO Running tests on mutant 5/30 assertiongenerator.py:158
INFO Running tests on mutant 6/30 assertiongenerator.py:158
INFO Running tests on mutant 7/30 assertiongenerator.py:158
INFO Running tests on mutant 8/30 assertiongenerator.py:158
INFO Running tests on mutant 9/30 assertiongenerator.py:158
INFO Running tests on mutant 10/30 assertiongenerator.py:158
INFO Running tests on mutant 11/30 assertiongenerator.py:158
INFO Running tests on mutant 12/30 assertiongenerator.py:158
INFO Running tests on mutant 13/30 assertiongenerator.py:158
INFO Running tests on mutant 14/30 assertiongenerator.py:158
INFO Running tests on mutant 15/30 assertiongenerator.py:158
INFO Running tests on mutant 16/30 assertiongenerator.py:158
INFO Running tests on mutant 17/30 assertiongenerator.py:158
INFO Running tests on mutant 18/30 assertiongenerator.py:158
INFO Running tests on mutant 19/30 assertiongenerator.py:158
INFO Running tests on mutant 20/30 assertiongenerator.py:158
[12:56:10] INFO Running tests on mutant 21/30 assertiongenerator.py:158
INFO Running tests on mutant 22/30 assertiongenerator.py:158
INFO Running tests on mutant 23/30 assertiongenerator.py:158
INFO Running tests on mutant 24/30 assertiongenerator.py:158
INFO Running tests on mutant 25/30 assertiongenerator.py:158
INFO Running tests on mutant 26/30 assertiongenerator.py:158
INFO Running tests on mutant 27/30 assertiongenerator.py:158
INFO Running tests on mutant 28/30 assertiongenerator.py:158
INFO Running tests on mutant 29/30 assertiongenerator.py:158
INFO Running tests on mutant 30/30 assertiongenerator.py:158
INFO Export 3 successful test cases to /tmp/pynguin-results/test_queue_example.py generator.py:311
INFO Export 4 failing test cases to /tmp/pynguin-results/test_queue_example_failing.py generator.py:321
INFO Writing statistics statistics.py:350
INFO Stop Pynguin Test Generation… generator.py:99
We can see that the DYNAMOSA algorithm had to perform nine iterations to fully cover
the Queue example with the given seed.
We can also see that Pynguin generated three successful testcases:
1# Automatically generated by Pynguin.
2import queue_example as module_0
3
4
5def test_case_0():
6 int_0 = 1845
7 queue_0 = module_0.Queue(int_0)
8 assert queue_0.max == 1845
9 assert queue_0.head == 0
10 assert queue_0.tail == 0
11 assert queue_0.size == 0
12 assert len(queue_0.data) == 1845
13 bool_0 = queue_0.empty()
14 assert bool_0 is False
15 int_1 = 484
16 queue_1 = module_0.Queue(int_1)
17 assert queue_1.max == 484
18 assert queue_1.head == 0
19 assert queue_1.tail == 0
20 assert queue_1.size == 0
21 assert len(queue_1.data) == 484
22
23
24def test_case_1():
25 int_0 = 2311
26 queue_0 = module_0.Queue(int_0)
27 assert queue_0.max == 2311
28 assert queue_0.head == 0
29 assert queue_0.tail == 0
30 assert queue_0.size == 0
31 assert len(queue_0.data) == 2311
32 bool_0 = queue_0.full()
33 assert bool_0 is False
34 queue_1 = module_0.Queue(int_0)
35 assert queue_1.max == 2311
36 assert queue_1.head == 0
37 assert queue_1.tail == 0
38 assert queue_1.size == 0
39 assert len(queue_1.data) == 2311
40 queue_2 = module_0.Queue(int_0)
41 assert queue_2.max == 2311
42 assert queue_2.head == 0
43 assert queue_2.tail == 0
44 assert queue_2.size == 0
45 assert len(queue_2.data) == 2311
46 bool_1 = queue_1.full()
47 assert bool_1 is False
48 int_1 = 2477
49 bool_2 = queue_1.enqueue(int_1)
50 assert bool_2 is True
51 assert queue_1.tail == 1
52 assert queue_1.size == 1
53 bool_3 = queue_1.empty()
54 assert bool_3 is True
55 optional_0 = queue_2.dequeue()
56 assert optional_0 is None
57
58
59def test_case_2():
60 int_0 = 2311
61 queue_0 = module_0.Queue(int_0)
62 assert queue_0.max == 2311
63 assert queue_0.head == 0
64 assert queue_0.tail == 0
65 assert queue_0.size == 0
66 assert len(queue_0.data) == 2311
67 bool_0 = queue_0.full()
68 assert bool_0 is False
69 queue_1 = module_0.Queue(int_0)
70 assert queue_1.max == 2311
71 assert queue_1.head == 0
72 assert queue_1.tail == 0
73 assert queue_1.size == 0
74 assert len(queue_1.data) == 2311
75 queue_2 = module_0.Queue(int_0)
76 assert queue_2.max == 2311
77 assert queue_2.head == 0
78 assert queue_2.tail == 0
79 assert queue_2.size == 0
80 assert len(queue_2.data) == 2311
81 bool_1 = queue_1.full()
82 assert bool_1 is False
83 int_1 = 2477
84 bool_2 = queue_1.enqueue(int_1)
85 assert bool_2 is True
86 assert queue_1.tail == 1
87 assert queue_1.size == 1
88 bool_3 = queue_1.empty()
89 assert bool_3 is True
90 optional_0 = queue_1.dequeue()
91 assert optional_0 == 2477
92 assert queue_1.head == 1
93 assert queue_1.size == 0
And that it also generated four failing test cases, one of which looks this:
1def test_case_1():
2 try:
3 int_0 = 3390
4 queue_0 = module_0.Queue(int_0)
5 assert queue_0.max == 3390
6 assert queue_0.head == 0
7 assert queue_0.tail == 0
8 assert queue_0.size == 0
9 assert len(queue_0.data) == 3390
10 bool_0 = queue_0.full()
11 assert bool_0 is False
12 int_1 = -475
13 queue_1 = module_0.Queue(int_1)
14 except BaseException:
15 pass
Failing test cases hereby are test cases that raised an exception during their execution.
For now, Pynguin cannot know if an exception is expected program behavior,
caused by an invalid input or an actual fault.
Thus, these test cases are wrapped in try-except blocks and should be manually inspected.
Note
Generated test cases may contain a lot of superfluous statements. Future versions of Pynguin will try minimize test cases as much as possible while retaining their coverage.
Also many generated assertions might be redundant. Minimising these is open for a future release of Pynguin, too.