Reference

pynguin.cli

Pynguin is an automated unit test generation framework for Python.

This module provides the main entry location for the program execution from the command line.

pynguin.cli.main(argv=None)

Entry point for the CLI of the Pynguin automatic unit test generation framework.

This method behaves like a standard UNIX command-line application, i.e., the return value 0 signals a successful execution. Any other return value signals some errors. This is, e.g., the case if the framework was not able to generate one successfully running test case for the class under test.

Parameters

argv – List of command-line arguments

Returns

An integer representing the success of the program run. 0 means success, all non-zero exit codes indicate errors.

pynguin.configuration

Provides a configuration interface for the test generator.

class pynguin.configuration.Algorithm(value)

Different algorithms supported by Pynguin.

DYNAMOSA = 'DYNAMOSA'

The dynamic many-objective sorting algorithm (cf. Panichella et al. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. TSE vol. 44 issue 2).

MIO = 'MIO'

The MIO test suite generation algorithm (cf. Andrea Arcuri. Many Independent Objective (MIO) Algorithm for Test Suite Generation. Proc. SBSE 2017).

MOSA = 'MOSA'

The many-objective sorting algorithm (cf. Panichella et al. Reformulating Branch Coverage as a Many-Objective Optimization Problem. Proc. ICST 2015).

RANDOM = 'RANDOM'

A feedback-direct random test generation approach similar to the algorithm proposed by Randoop (cf. Pacheco et al. Feedback-directed random test generation. Proc. ICSE 2007).

Performs random search on test cases.

Performs random search on test suites.

WHOLE_SUITE = 'WHOLE_SUITE'

A whole-suite test generation approach similar to the one proposed by EvoSuite (cf. Fraser and Arcuri. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. Proc. ESEC/FSE 2011).

This algorithm can be modified to use an archive (cf. Rojas, José Miguel, et al. “A detailed investigation of the effectiveness of whole test suite generation.” Empirical Software Engineering 22.2 (2017): 852-893.), by using the following options: –use-archive True, –seed_from_archive True and –filter_covered_targets_from_test_cluster True.

class pynguin.configuration.AssertionGenerator(value)

Different approaches for assertion generation supported by Pynguin.

MUTATION_ANALYSIS = 'MUTATION_ANALYSIS'

Use the mutation analysis approach for assertion generation.

NONE = 'NONE'

Do not create any assertions.

SIMPLE = 'SIMPLE'

Use the simple approach for primitive and none assertion generation.

class pynguin.configuration.Configuration(project_path, module_name, test_case_output, algorithm=Algorithm.DYNAMOSA, statistics_output=<factory>, stopping=<factory>, seeding=<factory>, type_inference=<factory>, test_creation=<factory>, search_algorithm=<factory>, mio=<factory>, random=<factory>)

General configuration for the test generator.

algorithm: pynguin.configuration.Algorithm = 'DYNAMOSA'

The algorithm that shall be used for generation.

mio: pynguin.configuration.MIOConfiguration

Configuration used for the MIO algorithm.

module_name: str

Name of the module for which the generator shall create tests.

project_path: str

Path to the project the generator shall create tests for.

random: pynguin.configuration.RandomConfiguration

Configuration used for the RANDOM algorithm.

search_algorithm: pynguin.configuration.SearchAlgorithmConfiguration

Search algorithm configuration.

seeding: pynguin.configuration.SeedingConfiguration

Seeding configuration.

statistics_output: pynguin.configuration.StatisticsOutputConfiguration

Statistic Output configuration.

stopping: pynguin.configuration.StoppingConfiguration

Stopping configuration.

test_case_output: pynguin.configuration.TestCaseOutputConfiguration

Configuration for how test cases should be output.

test_creation: pynguin.configuration.TestCreationConfiguration

Test creation configuration.

type_inference: pynguin.configuration.TypeInferenceConfiguration

Type inference configuration.

class pynguin.configuration.ExportStrategy(value)

Contains all available export strategies.

These strategies allow to export the generated test cases in different styles, such as the style of the PyTest framework. Setting the value to NONE will prevent exporting of the generated test cases (only reasonable for benchmarking, though).

NONE = 'NONE'

Do not export test cases at all.

PY_TEST = 'PY_TEST'

Export tests in the style of the PyTest framework.

class pynguin.configuration.MIOConfiguration(initial_config=<factory>, focused_config=<factory>, exploitation_starts_at_percent=0.5)

Configuration that is specific to the MIO approach.

exploitation_starts_at_percent: float = 0.5

Percentage ]0,1] of search budget after which exploitation is activated, i.e., switching to focused phase.

focused_config: pynguin.configuration.MIOPhaseConfiguration

Configuration to use in focused phase

initial_config: pynguin.configuration.MIOPhaseConfiguration

Configuration to use before focused phase.

class pynguin.configuration.MIOPhaseConfiguration(number_of_tests_per_target, random_test_or_from_archive_probability, number_of_mutations)

Configuration for a phase of MIO.

number_of_mutations: int

Number of mutations allowed to be done on the same individual before sampling a new one.

number_of_tests_per_target: int

Number of test cases for each target goal to keep in an archive.

random_test_or_from_archive_probability: float

Probability [0,1] of sampling a new test at random or choose an existing one in an archive.

class pynguin.configuration.MutationStrategy(value)

Different strategies for creating mutants when using the MUTATION_ANALYSIS approach for assertion generation.

BETWEEN_OPERATORS = 'BETWEEN_OPERATORS'

Higher order mutation strategy BetweenOperators. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)

EACH_CHOICE = 'EACH_CHOICE'

Higher order mutation strategy EachChoice. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)

FIRST_ORDER_MUTANTS = 'FIRST_ORDER_MUTANTS'

Generate first order mutants.

FIRST_TO_LAST = 'FIRST_TO_LAST'

Higher order mutation strategy FirstToLast. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)

RANDOM = 'RANDOM'

Higher order mutation strategy Random. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)

class pynguin.configuration.RandomConfiguration(max_sequence_length=10, max_sequences_combined=10)

Configuration that is specific to the RANDOM approach.

max_sequence_length: int = 10

The maximum length of sequences that are generated, 0 means infinite.

max_sequences_combined: int = 10

The maximum number of combined sequences, 0 means infinite.

class pynguin.configuration.SearchAlgorithmConfiguration(min_initial_tests=1, max_initial_tests=10, population=50, chromosome_length=40, chop_max_length=True, elite=1, crossover_rate=0.75, test_insertion_probability=0.1, test_delete_probability=0.3333333333333333, test_change_probability=0.3333333333333333, test_insert_probability=0.3333333333333333, statement_insertion_probability=0.5, random_perturbation=0.2, change_parameter_probability=0.1, tournament_size=5, rank_bias=1.7, selection=Selection.TOURNAMENT_SELECTION, use_archive=False, filter_covered_targets_from_test_cluster=False)

General configuration for search algorithms.

change_parameter_probability: float = 0.1

Probability of replacing parameters when mutating a method or constructor statement in a test case. Expects values in [0,1]

chop_max_length: bool = True

Chop statements after exception if length has reached maximum

chromosome_length: int = 40

Maximum length of chromosomes during search

crossover_rate: float = 0.75

Probability of crossover

elite: int = 1

Elite size for search algorithm

filter_covered_targets_from_test_cluster: bool = False

Focus search by filtering out elements from the test cluster when they are fully covered.

max_initial_tests: int = 10

Maximum number of tests in initial test suites

min_initial_tests: int = 1

Minimum number of tests in initial test suites

population: int = 50

Population size of genetic algorithm

random_perturbation: float = 0.2

Probability to replace a primitive with a random new value rather than adding a delta.

rank_bias: float = 1.7

Bias for better individuals in rank selection

selection: pynguin.configuration.Selection = 'TOURNAMENT_SELECTION'

The selection operator for genetic algorithms.

statement_insertion_probability: float = 0.5

Initial probability of inserting a new statement in a test case

test_change_probability: float = 0.3333333333333333

Probability of changing statements during mutation

test_delete_probability: float = 0.3333333333333333

Probability of deleting statements during mutation

test_insert_probability: float = 0.3333333333333333

Probability of inserting new statements during mutation

test_insertion_probability: float = 0.1

Initial probability of inserting a new test in a test suite

tournament_size: int = 5

Number of individuals for tournament selection.

use_archive: bool = False

Some algorithms can be enhanced with an optional archive, e.g. Whole Suite -> Whole Suite + Archive. Use this option to enable the usage of an archive. Algorithms that always use an archive are not affected by this option.

class pynguin.configuration.SeedingConfiguration(seed=None, constant_seeding=True, initial_population_seeding=False, initial_population_data='', seeded_testcases_reuse_probability=0.9, initial_population_mutations=0, dynamic_constant_seeding=True, seeded_primitives_reuse_probability=0.2, seeded_dynamic_values_reuse_probability=0.6, seed_from_archive=False, seed_from_archive_probability=0.2, seed_from_archive_mutations=3)

Configuration related to seeding.

constant_seeding: bool = True

Should the generator use a static constant seeding technique to improve constant generation?

dynamic_constant_seeding: bool = True

Enables seeding of constants at runtime.

initial_population_data: str = ''

The path to the file with the pre-existing tests. The path has to include the file itself.

initial_population_mutations: int = 0

Number of how often the testcases collected by initial population seeding should be mutated to promote diversity

initial_population_seeding: bool = False

Should the generator use previously existing testcases to seed the initial population?

seed: Optional[int] = None

A predefined seed value for the random number generator that is used.

seed_from_archive: bool = False

When sampling new test cases reuse some from the archive, if one is used.

seed_from_archive_mutations: int = 3

Number of mutations applied when sampling from the archive.

seed_from_archive_probability: float = 0.2

Instead of creating a new test case, reuse a covering solution from the archive, iff an archive is used.

seeded_dynamic_values_reuse_probability: float = 0.6

Probability of using dynamically seeded values when a primitive seeded value will be used.

seeded_primitives_reuse_probability: float = 0.2

Probability for using seeded primitive values instead of randomly generated ones.

seeded_testcases_reuse_probability: float = 0.9

Probability of using seeded testcases when initial population seeding is enabled.

class pynguin.configuration.Selection(value)

Different selection algorithms to select from.

RANK_SELECTION = 'RANK_SELECTION'

Rank selection.

TOURNAMENT_SELECTION = 'TOURNAMENT_SELECTION'

Tournament selection. Use tournament_size to set size.

class pynguin.configuration.StatisticsBackend(value)

The different available statistics backends to write statistics

CONSOLE = 'CONSOLE'

Write statistics to the standard out.

CSV = 'CSV'

Write statistics to a CSV file.

NONE = 'NONE'

Do not write any statistics.

class pynguin.configuration.StatisticsOutputConfiguration(report_dir='pynguin-report', statistics_backend=StatisticsBackend.CSV, timeline_interval=1000000000, timeline_interpolation=True, output_variables=<factory>, configuration_id='', project_name='', create_coverage_report=False)

Configuration related to output.

configuration_id: str = ''

Label that identifies the used configuration of Pynguin. This is only done when running experiments.

create_coverage_report: bool = False

Create a coverage report for the tested module. This can be helpful to find hard to cover parts because Pynguin measures coverage on bytecode level which might yield different results when compared with other tools, e.g., Coverage.py.

output_variables: List[pynguin.utils.statistics.runtimevariable.RuntimeVariable]

List of variables to output to the statistics backend.

project_name: str = ''

Label that identifies the project name of Pynguin. This is useful when running experiments.

report_dir: str = 'pynguin-report'

Directory in which to put HTML and CSV reports

statistics_backend: pynguin.configuration.StatisticsBackend = 'CSV'

Which backend to use to collect data

timeline_interpolation: bool = True

Interpolate timeline values

timeline_interval: int = 1000000000

Time interval in nano-seconds for timeline statistics, i.e., we select a data point after each interval. This can be interpolated, if there is no exact value stored at the time-step of the interval, see timeline_interpolation. The default value is every 1.00s.

class pynguin.configuration.StoppingCondition(value)

The different stopping conditions for the algorithms.

MAX_ITERATIONS = 'MAX_ITERATIONS'

Stop after a maximum number of algorithm iterations.

MAX_STATEMENT_EXECUTIONS = 'MAX_STATEMENT_EXECUTIONS'

Stop as soon as a maximum number of statements were executed.

MAX_TEST_EXECUTIONS = 'MAX_TEST_EXECUTIONS'

Stop as soon as a maximum number of tests were executed.

MAX_TIME = 'MAX_TIME'

Stop after a maximum time limit has been reached.

class pynguin.configuration.StoppingConfiguration(stopping_condition=StoppingCondition.MAX_TIME, budget=600, maximum_test_executions=60, maximum_statement_executions=600, maximum_iterations=60)

Configuration related to when Pynguin should stop.

budget: int = 600

Time budget (in seconds) that can be used for generating tests.

maximum_iterations: int = 60

Maximum iterations

maximum_statement_executions: int = 600

Maximum number of test cases to be executed.

maximum_test_executions: int = 60

Maximum number of test cases to be executed.

stopping_condition: pynguin.configuration.StoppingCondition = 'MAX_TIME'

What condition should be checked to end the search/test generation.

class pynguin.configuration.TestCaseOutputConfiguration(output_path, export_strategy=ExportStrategy.PY_TEST, max_length_test_case=2500, assertion_generation=AssertionGenerator.MUTATION_ANALYSIS, allow_stale_assertions=False, mutation_strategy=MutationStrategy.FIRST_ORDER_MUTANTS, mutation_order=1, post_process=True, float_precision=0.01)

Configuration related to test case output.

allow_stale_assertions: bool = False

Allow assertion on things that did not change between statement executions.

assertion_generation: pynguin.configuration.AssertionGenerator = 'MUTATION_ANALYSIS'

The generator that shall be used for assertion generation.

export_strategy: pynguin.configuration.ExportStrategy = 'PY_TEST'

The export strategy determines for which test-runner system the generated tests should fit.

float_precision: float = 0.01

Precision to use in float comparisons and assertions

max_length_test_case: int = 2500

The maximum number of statement in as test case (normal + assertion statements)

mutation_order: int = 1

The order of the generated higher order mutants in the mutation analysis assertion generation method.

mutation_strategy: pynguin.configuration.MutationStrategy = 'FIRST_ORDER_MUTANTS'

The strategy that shall be used for creating mutants in the mutation analysis assertion generation method.

output_path: str

Path to an output folder for the generated test cases.

post_process: bool = True

Should the results be post processed? For example, truncate test cases after statements that raise an exception.

class pynguin.configuration.TestCreationConfiguration(max_recursion=10, max_delta=20, max_int=2048, string_length=20, bytes_length=20, collection_size=5, primitive_reuse_probability=0.5, object_reuse_probability=0.9, none_probability=0.1, skip_optional_parameter_probability=0.7, max_attempts=1000, insertion_uut=0.5, max_size=100)

Configuration related to test creation.

bytes_length: int = 20

Maximum length of randomly generated bytes

collection_size: int = 5

Maximum length of randomly generated collections

insertion_uut: float = 0.5

Score for selection of insertion of UUT calls

max_attempts: int = 1000

Number of attempts when generating an object before giving up

max_delta: int = 20

Maximum size of delta for numbers during mutation

max_int: int = 2048

Maximum size of randomly generated integers (minimum range = -1 * max)

max_recursion: int = 10

Recursion depth when trying to create objects in a test case.

max_size: int = 100

Maximum number of test cases in a test suite

none_probability: float = 0.1

Probability to use None in a test case instead of constructing an object. Expects values in [0,1]

object_reuse_probability: float = 0.9

Probability to reuse an existing object in a test case, if available. Expects values in [0,1]

primitive_reuse_probability: float = 0.5

Probability to reuse an existing primitive in a test case, if available. Expects values in [0,1]

skip_optional_parameter_probability: float = 0.7

Probability to skip an optional parameter, i.e., do not fill this parameter.

string_length: int = 20

Maximum length of randomly generated strings

class pynguin.configuration.TypeInferenceConfiguration(guess_unknown_types=True, type_inference_strategy=TypeInferenceStrategy.TYPE_HINTS, max_cluster_recursion=10, stub_dir=None)

Configuration related to type inference.

guess_unknown_types: bool = True

Should we guess unknown types while constructing parameters? This might happen in the following cases: The parameter type is unknown, e.g. a parameter is missing a type hint. The parameter is not primitive and cannot be created from the test cluster, e.g. Callable[…]

max_cluster_recursion: int = 10

The maximum level of recursion when calculating the dependencies in the test cluster.

stub_dir: Optional[str] = None

Path to the pyi-stub files for the StubInferenceStrategy

type_inference_strategy: pynguin.configuration.TypeInferenceStrategy = 'TYPE_HINTS'

The strategy for type-inference that shall be used

class pynguin.configuration.TypeInferenceStrategy(value)

The different available type-inference strategies.

NONE = 'NONE'

Ignore any type information given in the module under test.

STUB_FILES = 'STUB_FILES'

Use type information from stub files.

TYPE_HINTS = 'TYPE_HINTS'

Use type information from type hints in the module under test.

pynguin.generator

Pynguin is an automated unit test generation framework for Python.

The framework generates unit tests for a given Python module. For this it supports various approaches, such as a random approach, similar to Randoop or a whole-suite approach, based on a genetic algorithm, as implemented in EvoSuite. The framework allows to export test suites in various styles, i.e., using the unittest library from the Python standard library or tests in the style used by the PyTest framework.

Pynguin is supposed to be used as a standalone command-line application but it can also be used as a library by instantiating this class directly.

class pynguin.generator.ReturnCode(value)

Return codes for Pynguin to signal result.

NO_TESTS_GENERATED = 2

Symbolises that no test could be generated.

OK = 0

Symbolises that the execution ended as expected.

SETUP_FAILED = 1

Symbolises that the execution failed in the setup phase.

pynguin.generator.run_pynguin()

Run the test generation.

The result of the test generation is indicated by the resulting ReturnCode.

Return type

ReturnCode

Returns

See ReturnCode.

Raises

ConfigurationException – In case the configuration is illegal

pynguin.generator.set_configuration(configuration)

Initialises the test generator with the given configuration.

Parameters

configuration (Configuration) – The configuration to use.

Return type

None