Reproducible runs

According to the Python documentation it is necessary to set the environment variable PYTHONHASHSEED=0 in order to achieve a truly deterministic behaviour and computation. Furthermore, it is necessary to set the --seed command-line option to a fixed value. If no --seed option was set, Pynguin chooses a seed at random and logs its value.

By default, Pynguin will search for up to ten minutes of wall clock time. You can change this behaviour by specifying one or multiple stopping conditions to stop after a certain amount of

  • wall clock time (--maximum-search-time)

  • executed test cases (--maximum-test-executions)

  • executed statements (--maximum-statement-executions)

  • algorithm iterations (--maximum-iterations)

Pynguin will stop, as soon as one of the stopping conditions is fulfilled or when Pynguin was able to cover everything in the module under test w.r.t. the specified coverage metrics.

Note

Using a certain amount of wall-clock time as the stopping condition (--maximum-search-time) for Pynguin’s executions can make achieving deterministic results difficult, because Pynguin might achieve slightly different amounts of iterations in each execution. To solve this problem, take note of the amount of achieved iterations in the first execution and use them as the new stopping condition (--maximum-iterations) in a reproduction run, together with the previously used seed. The amount of iterations can be obtained from stdout (when using -v for verbose output) or from the tracked statistics, when tracking the variable AlgorithmIterations.

Note

Pynguin’s algorithms check the stopping condition at the beginning of each iteration, i.e., if the stopping condition is reached within an interation, this iteration will still be finished.