Pynguin is an automated unit test generation framework for Python.
This module provides the main entry location for the program execution from the command line.
Entry point for the CLI of the Pynguin automatic unit test generation framework.
This method behaves like a standard UNIX command-line application, i.e., the return value 0 signals a successful execution. Any other return value signals some errors. This is, e.g., the case if the framework was not able to generate one successfully running test case for the class under test.
Provides a configuration interface for the test generator.
- class pynguin.configuration.Algorithm(value)
Different algorithms supported by Pynguin.
- DYNAMOSA = 'DYNAMOSA'
The dynamic many-objective sorting algorithm (cf. Panichella et al. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. TSE vol. 44 issue 2).
- MIO = 'MIO'
The MIO test suite generation algorithm (cf. Andrea Arcuri. Many Independent Objective (MIO) Algorithm for Test Suite Generation. Proc. SBSE 2017).
- MOSA = 'MOSA'
The many-objective sorting algorithm (cf. Panichella et al. Reformulating Branch Coverage as a Many-Objective Optimization Problem. Proc. ICST 2015).
- RANDOM = 'RANDOM'
A feedback-direct random test generation approach similar to the algorithm proposed by Randoop (cf. Pacheco et al. Feedback-directed random test generation. Proc. ICSE 2007).
- RANDOM_TEST_CASE_SEARCH = 'RANDOM_TEST_CASE_SEARCH'
Performs random search on test cases.
- RANDOM_TEST_SUITE_SEARCH = 'RANDOM_TEST_SUITE_SEARCH'
Performs random search on test suites.
- WHOLE_SUITE = 'WHOLE_SUITE'
A whole-suite test generation approach similar to the one proposed by EvoSuite (cf. Fraser and Arcuri. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. Proc. ESEC/FSE 2011).
This algorithm can be modified to use an archive (cf. Rojas, José Miguel, et al. “A detailed investigation of the effectiveness of whole test suite generation.” Empirical Software Engineering 22.2 (2017): 852-893.), by using the following options: –use-archive True, –seed-from-archive True and –filter-covered-targets-from-test-cluster True.
- class pynguin.configuration.AssertionGenerator(value)
Different approaches for assertion generation supported by Pynguin.
- CHECKED_MINIMIZING = 'CHECKED_MINIMIZING'
All assertions that do not increase the checked coverage are removed.
- MUTATION_ANALYSIS = 'MUTATION_ANALYSIS'
Use the mutation analysis approach for assertion generation.
- NONE = 'NONE'
Do not create any assertions.
- SIMPLE = 'SIMPLE'
Use the simple approach for primitive and none assertion generation.
- class pynguin.configuration.Configuration(project_path, module_name, test_case_output, algorithm=Algorithm.DYNAMOSA, statistics_output=<factory>, stopping=<factory>, seeding=<factory>, type_inference=<factory>, test_creation=<factory>, search_algorithm=<factory>, mio=<factory>, random=<factory>)
General configuration for the test generator.
- class pynguin.configuration.CoverageMetric(value)
The different available coverage metrics available for optimisation.
- BRANCH = 'BRANCH'
Calculate how many of the possible branches in the code were executed
- CHECKED = 'CHECKED'
Calculate how many of the possible lines in the code are checked by an assertion.
- LINE = 'LINE'
Calculate how many of the possible lines in the code were executed
- class pynguin.configuration.ExportStrategy(value)
Contains all available export strategies.
These strategies allow to export the generated test cases in different styles, such as the style of the PyTest framework. Setting the value to NONE will prevent exporting of the generated test cases (only reasonable for benchmarking, though).
- NONE = 'NONE'
Do not export test cases at all.
- PY_TEST = 'PY_TEST'
Export tests in the style of the PyTest framework.
- class pynguin.configuration.MIOConfiguration(initial_config=<factory>, focused_config=<factory>, exploitation_starts_at_percent=0.5)
Configuration that is specific to the MIO approach.
Percentage ]0,1] of search budget after which exploitation is activated, i.e., switching to focused phase.
- class pynguin.configuration.MIOPhaseConfiguration(number_of_tests_per_target, random_test_or_from_archive_probability, number_of_mutations)
Configuration for a phase of MIO.
Number of mutations allowed to be done on the same individual before sampling a new one.
- class pynguin.configuration.MutationStrategy(value)
Different strategies for creating mutants.
Only respected when using the MUTATION_ANALYSIS approach for assertion generation.
- BETWEEN_OPERATORS = 'BETWEEN_OPERATORS'
Higher order mutation strategy BetweenOperators. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)
- EACH_CHOICE = 'EACH_CHOICE'
Higher order mutation strategy EachChoice. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)
- FIRST_ORDER_MUTANTS = 'FIRST_ORDER_MUTANTS'
Generate first order mutants.
- FIRST_TO_LAST = 'FIRST_TO_LAST'
Higher order mutation strategy FirstToLast. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)
- RANDOM = 'RANDOM'
Higher order mutation strategy Random. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)
- class pynguin.configuration.RandomConfiguration(max_sequence_length=10, max_sequences_combined=10)
Configuration that is specific to the RANDOM approach.
The maximum length of sequences that are generated, 0 means infinite.
- class pynguin.configuration.SearchAlgorithmConfiguration(min_initial_tests=1, max_initial_tests=10, population=50, chromosome_length=40, chop_max_length=True, elite=1, crossover_rate=0.75, test_insertion_probability=0.1, test_delete_probability=0.3333333333333333, test_change_probability=0.3333333333333333, test_insert_probability=0.3333333333333333, statement_insertion_probability=0.5, random_perturbation=0.2, change_parameter_probability=0.1, tournament_size=5, rank_bias=1.7, selection=Selection.TOURNAMENT_SELECTION, use_archive=False, filter_covered_targets_from_test_cluster=False)
General configuration for search algorithms.
Probability of replacing parameters when mutating a method or constructor statement in a test case. Expects values in [0,1]
Focus search by filtering out elements from the test cluster when they are fully covered.
Probability to replace a primitive with a random new value rather than adding a delta.
Initial probability of inserting a new statement in a test case
Probability of changing statements during mutation
Probability of deleting statements during mutation
Probability of inserting new statements during mutation
Initial probability of inserting a new test in a test suite
- class pynguin.configuration.SeedingConfiguration(seed=1693475369381198021, constant_seeding=True, initial_population_seeding=False, initial_population_data='', seeded_testcases_reuse_probability=0.9, initial_population_mutations=0, dynamic_constant_seeding=True, seeded_primitives_reuse_probability=0.2, seeded_dynamic_values_reuse_probability=0.6, seed_from_archive=False, seed_from_archive_probability=0.2, seed_from_archive_mutations=3, max_dynamic_length=1000, max_dynamic_pool_size=50)
Configuration related to seeding.
Should the generator use a static constant seeding technique to improve constant generation?
The path to the file with the pre-existing tests. The path has to include the file itself.
Number of how often the testcases collected by initial population seeding should be mutated to promote diversity
Should the generator use previously existing testcases to seed the initial population?
Maximum length of strings/bytes that should be stored in the dynamic constant pool.
Maximum number of constants of the same type that should be stored in the dynamic constant pool.
A predefined seed value for the random number generator that is used.
When sampling new test cases reuse some from the archive, if one is used.
Instead of creating a new test case, reuse a covering solution from the archive, iff an archive is used.
Probability of using dynamically seeded values when a primitive seeded value will be used.
Probability for using seeded primitive values instead of randomly generated ones.
- class pynguin.configuration.Selection(value)
Different selection algorithms to select from.
- RANK_SELECTION = 'RANK_SELECTION'
- TOURNAMENT_SELECTION = 'TOURNAMENT_SELECTION'
Tournament selection. Use tournament_size to set size.
- class pynguin.configuration.StatisticsBackend(value)
The different available statistics backends to write statistics.
- CONSOLE = 'CONSOLE'
Write statistics to the standard out.
- CSV = 'CSV'
Write statistics to a CSV file.
- NONE = 'NONE'
Do not write any statistics.
- class pynguin.configuration.StatisticsOutputConfiguration(report_dir='pynguin-report', statistics_backend=StatisticsBackend.CSV, timeline_interval=1000000000, timeline_interpolation=True, coverage_metrics=<factory>, output_variables=<factory>, configuration_id='', run_id='', project_name='', create_coverage_report=False, type_guess_top_n=10)
Configuration related to output.
Label that identifies the used configuration of Pynguin. This is only done when running experiments.
List of coverage metrics that are optimised during the search
Create a coverage report for the tested module. This can be helpful to find hard to cover parts because Pynguin measures coverage on bytecode level which might yield different results when compared with other tools, e.g., Coverage.py.
Label that identifies the project name of Pynguin. This is useful when running experiments.
Id of the cluster run. Useful for finding the log entries that belong to a certain result.
Time interval in nano-seconds for timeline statistics, i.e., we select a data point after each interval. This can be interpolated, if there is no exact value stored at the time-step of the interval, see timeline_interpolation. The default value is every 1.00s.
- class pynguin.configuration.StoppingConfiguration(maximum_search_time=-1, maximum_test_executions=-1, maximum_statement_executions=-1, maximum_slicing_time=600, maximum_iterations=-1, maximum_test_execution_timeout=5, maximum_coverage=100, maximum_coverage_plateau=-1, minimum_coverage=100, minimum_plateau_iterations=-1, test_execution_time_per_statement=1)
Configuration related to when Pynguin should stop.
Note that these are mostly soft-limits rather than hard limits, because the search algorithms only check the condition at the start of each algorithm iteration.
The maximum percentage of coverage after which the generation shall stop.
Maximum number of algorithm iterations without coverage change before the algorithms stops.
The maximum time (in seconds) after which a test case times out.
Minimum coverage for the plateau-based stopping condition. Expects values larger than 0 but less than 100 to activate the stopping condition; also requires the setting of minimum_plateau_iterations.
Minimum iterations without a coverage change to stop early. Expects values larger than 0; also requires the setting of minimum_coverage.
- class pynguin.configuration.TestCaseOutputConfiguration(output_path, export_strategy=ExportStrategy.PY_TEST, max_length_test_case=2500, assertion_generation=AssertionGenerator.MUTATION_ANALYSIS, allow_stale_assertions=False, mutation_strategy=MutationStrategy.FIRST_ORDER_MUTANTS, mutation_order=1, post_process=True, float_precision=0.01, format_with_black=True)
Configuration related to test case output.
Allow assertion on things that did not change between statement executions.
The generator that shall be used for assertion generation.
The export strategy determines for which test-runner system the generated tests should fit.
The maximum number of statement in as test case (normal + assertion statements)
The order of the generated higher order mutants in the mutation analysis assertion generation method.
The strategy that shall be used for creating mutants in the mutation analysis assertion generation method.
- class pynguin.configuration.TestCreationConfiguration(max_recursion=10, max_delta=20, max_int=2048, string_length=20, bytes_length=20, collection_size=5, primitive_reuse_probability=0.5, object_reuse_probability=0.9, none_weight=1, any_weight=5, original_type_weight=5, type_tracing_weight=10, type4py_weight=10, type_tracing_kept_guesses=2, wrap_var_param_type_probability=0.7, negate_type=0.1, skip_optional_parameter_probability=0.7, max_attempts=1000, insertion_uut=0.5, max_size=100, use_random_object_for_call=0.1)
Configuration related to test creation.
Weight to use Any as parameter type during test generation. Expects values > 0.
When inferring a type from proxies, it may also be desirable to negate the chosen type, e.g., such that an instance check or a getattr() evaluate to False. Expects values in [0,1]
Weight to use None as parameter type during test generation. Expects values > 0.
Probability to reuse an existing object in a test case, if available. Expects values in [0,1]
Weight to use the originally annotated type as parameter type during test generation. Expects values > 0.
Probability to reuse an existing primitive in a test case, if available. Expects values in [0,1]
Probability to skip an optional parameter, i.e., do not fill such a parameter.
Weight to use types inferred from type4py as parameter type during test generation. Expects values > 0.
Amount of kept recently guessed types per parameter, when type tracing is used.
Weight to use the type guessed from type tracing as parameter type during test generation. Expects values > 0.
When adding or modifying a call on an object, use a random modifier instead of only modifiers for that type. Expects values in [0, 1].
- class pynguin.configuration.TypeInferenceConfiguration(type_inference_strategy=TypeInferenceStrategy.TYPE_HINTS, type_tracing=False, type4py=False, type4py_uri='https://type4py.com/', type4py_timeout=10)
Configuration related to type inference.
The strategy for type-inference that shall be used
Pynguin is an automated unit test generation framework for Python.
The framework generates unit tests for a given Python module. For this it supports various approaches, such as a random approach, similar to Randoop or a whole-suite approach, based on a genetic algorithm, as implemented in EvoSuite. The framework allows to export test suites in various styles, i.e., using the unittest library from the Python standard library or tests in the style used by the PyTest framework.
Pynguin is supposed to be used as a standalone command-line application but it can also be used as a library by instantiating this class directly.
- class pynguin.generator.ReturnCode(value)
Return codes for Pynguin to signal result.
- NO_TESTS_GENERATED = 2
Symbolises that no test could be generated.
- OK = 0
Symbolises that the execution ended as expected.
- SETUP_FAILED = 1
Symbolises that the execution failed in the setup phase.
Run the test generation.
The result of the test generation is indicated by the resulting ReturnCode.
- Return type:
ConfigurationException – In case the configuration is illegal