Reference

pynguin.cli

Pynguin is an automated unit test generation framework for Python.

This module provides the main entry location for the program execution from the command line.

pynguin.cli.main(argv=None)

Entry point for the CLI of the Pynguin automatic unit test generation framework.

This method behaves like a standard UNIX command-line application, i.e., the return value 0 signals a successful execution. Any other return value signals some errors. This is, e.g., the case if the framework was not able to generate one successfully running test case for the class under test.

Parameters:

argv (Optional[list[str]]) – List of command-line arguments

Return type:

int

Returns:

An integer representing the success of the program run. 0 means success, all non-zero exit codes indicate errors.

pynguin.configuration

Provides a configuration interface for the test generator.

class pynguin.configuration.Algorithm(value)

Different algorithms supported by Pynguin.

DYNAMOSA = 'DYNAMOSA'

The dynamic many-objective sorting algorithm (cf. Panichella et al. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. TSE vol. 44 issue 2).

LLM = 'LLM'

This does not execute the usual test and assertion generation stages of Pynguin but queries an LLM and writes the resulting test cases to the output file.

Type:

Query a large language model for tests. NOTE

LLMOSA = 'LLMOSA'

The many-objective sorting algorithm with LLM.

MIO = 'MIO'

The MIO test suite generation algorithm (cf. Andrea Arcuri. Many Independent Objective (MIO) Algorithm for Test Suite Generation. Proc. SBSE 2017).

MOSA = 'MOSA'

The many-objective sorting algorithm (cf. Panichella et al. Reformulating Branch Coverage as a Many-Objective Optimization Problem. Proc. ICST 2015).

RANDOM = 'RANDOM'

A feedback-direct random test generation approach similar to the algorithm proposed by Randoop (cf. Pacheco et al. Feedback-directed random test generation. Proc. ICSE 2007).

RANDOM_TEST_CASE_SEARCH = 'RANDOM_TEST_CASE_SEARCH'

Performs random search on test cases.

RANDOM_TEST_SUITE_SEARCH = 'RANDOM_TEST_SUITE_SEARCH'

Performs random search on test suites.

WHOLE_SUITE = 'WHOLE_SUITE'

A whole-suite test generation approach similar to the one proposed by EvoSuite (cf. Fraser and Arcuri. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. Proc. ESEC/FSE 2011).

This algorithm can be modified to use an archive (cf. Rojas, José Miguel, et al. “A detailed investigation of the effectiveness of whole test suite generation.” Empirical Software Engineering 22.2 (2017): 852-893.), by using the following options: –use-archive True, –seed-from-archive True and –filter-covered-targets-from-test-cluster True.

class pynguin.configuration.AssertionGenerator(value)

Different approaches for assertion generation supported by Pynguin.

CHECKED_MINIMIZING = 'CHECKED_MINIMIZING'

All assertions that do not increase the checked coverage are removed.

LLM = 'LLM'

Use Large Language Model(LLM) for assertion generation.

MUTATION_ANALYSIS = 'MUTATION_ANALYSIS'

Use the mutation analysis approach for assertion generation.

NONE = 'NONE'

Do not create any assertions.

SIMPLE = 'SIMPLE'

Use the simple approach for primitive and none assertion generation.

class pynguin.configuration.Configuration(project_path, module_name, test_case_output, algorithm=Algorithm.DYNAMOSA, statistics_output=<factory>, stopping=<factory>, large_language_model=<factory>, string_statement=<factory>, seeding=<factory>, type_inference=<factory>, pynguinml=<factory>, test_creation=<factory>, generator_selection=<factory>, search_algorithm=<factory>, mio=<factory>, random=<factory>, to_cover=<factory>, llm_refinement=<factory>, ignore_modules=<factory>, ignore_methods=<factory>, element_visibility=ElementVisibility.PUBLIC, subprocess=False, subprocess_if_recommended=True, local_search=<factory>, use_master_worker=True, filesystem_isolation=False)

General configuration for the test generator.

algorithm: Algorithm = 'DYNAMOSA'

The algorithm that shall be used for generation.

element_visibility: ElementVisibility = 'PUBLIC'

Which elements (by naming convention) of the module under test are included in the test cluster: PUBLIC (default) only public elements, PROTECTED also elements with a single leading underscore, ALL also private (double leading underscore, including name-mangled) elements. Only applies to the module under test; non-public elements of dependency modules are always excluded.

filesystem_isolation: bool = False

Whether to use filesystem isolation during test execution. Filesystem isolation provides some safety by tracking and controlling file operations, e.g. preventing file deletion of non-created files, but adds performance overhead.

ignore_methods: list[str]

Ignore the methods specified here from the module analysis.

ignore_modules: list[str]

Ignore the modules specified here from the module analysis.

large_language_model: LLMConfiguration

Large Language Model(LLM) configuration.

llm_refinement: LLMRefinementConfiguration

LLM-based test refinement configuration.

local_search: LocalSearchConfiguration

Local search configuration.

mio: MIOConfiguration

Configuration used for the MIO algorithm.

module_name: str

Name of the module for which the generator shall create tests.

project_path: str

Path to the project the generator shall create tests for.

pynguinml: PynguinMLConfiguration

PynguinML configuration.

random: RandomConfiguration

Configuration used for the RANDOM algorithm.

search_algorithm: SearchAlgorithmConfiguration

Search algorithm configuration.

seeding: SeedingConfiguration

Seeding configuration.

statistics_output: StatisticsOutputConfiguration

Statistic Output configuration.

stopping: StoppingConfiguration

Stopping configuration.

string_statement: StringStatementConfiguration

String statement configuration.

subprocess: bool = False

Run the test generation in a subprocess.

subprocess_if_recommended: bool = True

Use the subprocess mode if the SUT analysis recommends it based on used C extension modules.

test_case_output: TestCaseOutputConfiguration

Configuration for how test cases should be output.

test_creation: TestCreationConfiguration

Test creation configuration.

to_cover: ToCoverConfiguration

Configuration of which code elements are included or excluded as coverage goals.

type_inference: TypeInferenceConfiguration

Type inference configuration.

use_master_worker: bool = True

Use master-worker architecture for fault tolerance.

class pynguin.configuration.CoverageMetric(value)

The different available coverage metrics available for optimisation.

BRANCH = 'BRANCH'

Calculate how many of the possible branches in the code were executed

CHECKED = 'CHECKED'

Calculate how many of the possible lines in the code are checked by an assertion.

LINE = 'LINE'

Calculate how many of the possible lines in the code were executed

class pynguin.configuration.ElementVisibility(value)

Which elements of a module are considered for test generation.

ALL = 'ALL'

Include public, protected, and private elements (double leading underscore, including name-mangled).

PROTECTED = 'PROTECTED'

Include public and protected elements (single leading underscore).

PUBLIC = 'PUBLIC'

Only include public elements (default, current behavior).

class pynguin.configuration.ExportStrategy(value)

Contains all available export strategies.

These strategies allow to export the generated test cases in different styles, such as the style of the PyTest framework. Setting the value to NONE will prevent exporting of the generated test cases (only reasonable for benchmarking, though).

NONE = 'NONE'

Do not export test cases at all.

PY_TEST = 'PY_TEST'

Export tests in the style of the PyTest framework.

class pynguin.configuration.GeneratorSelectionConfiguration(generator_selection_algorithm=Selection.RANK_SELECTION, generator_selection_bias=1.7, generator_any_distance=30, generator_not_constructor_penalty=10.0, generator_param_penalty=1.0, generator_hierarchy_penalty=1.0, generator_any_type_penalty=100.0)

Configuration related to selecting a generator for a certain type.

generator_any_distance: int = 30

The distance to and from the any type to any other type in the type hierarchy. As the any type is a super- and subtype of all type but it is most likely not the type we want, we want to punish it in the type selection process.

generator_any_type_penalty: float = 100.0

Penalty for selecting a generator that returns only Any. While Any is always a valid type, we still want to avoid it as a generator.

generator_hierarchy_penalty: float = 1.0

Penalty for the distance between the generators type and the desired type. The penalty is multiplied with the distance in the type hierarchy.

generator_not_constructor_penalty: float = 10.0

Penalty for selecting a generator that is not a __init__ method.

generator_param_penalty: float = 1.0

Penalty for each parameter of the generator.

generator_selection_algorithm: Selection = 'RANK_SELECTION'

The selection algorithm to use for selecting the generator for a statement. Currently only RANK_SELECTION is supported.

generator_selection_bias: float = 1.7

The bias to use for rank selection when selecting the generator for a statement. Expects values in [1.0, 2.0]. 1.0 is random selection, higher values bias towards higher ranked generators. A value above 2.0 leads to cutting off the lower ranked generators.

class pynguin.configuration.LLMConfiguration(api_key='', model_name='gpt-4o-mini', llm_url='', hybrid_initial_population=False, llm_test_case_percentage=0.5, enable_response_caching=False, call_llm_for_uncovered_targets=False, coverage_threshold=1, call_llm_on_stall_detection=False, stall_detection_window_seconds=30, max_plateau_len=25, max_llm_interventions=-1, min_remaining_budget_for_llm=45, max_context_chars=64000, max_retries=8, request_timeout=30.0, max_request_time=300.0, enable_thinking=False, cache_dir='~/.cache/pynguin/llm')

Configuration for the LLM.

api_key: str = ''

The api key to call OpenAI LLM with.

cache_dir: str = '~/.cache/pynguin/llm'

The directory to store cached responses.

call_llm_for_uncovered_targets: bool = False

Whether to call the LLM for the uncovered targets once, before the search iterations start. Like seeding, this front-loads token cost; prefer call_llm_on_stall_detection for the recommended stagnation-only setup.

call_llm_on_stall_detection: bool = False

Whether to query the LLM for uncovered targets when the search stalls (stagnation-triggered querying). This is the recommended integration mode: keep hybrid_initial_population and call_llm_for_uncovered_targets off and enable this so the GA covers easy goals first and the LLM is only spent to break through plateaus.

coverage_threshold: float = 1

The coverage threshold when to call the LLM for low-coverage targets. The value must be from [0.0, 1.0].

enable_response_caching: bool = False

Whether to enable caching for responses of the model.

enable_thinking: bool | None = False

Whether to request reasoning (“thinking”) output from a reasoning model, sent to the API as extra_body={"chat_template_kwargs": {"enable_thinking": ...}}. Defaults to False, suppressing reasoning output: reasoning models otherwise spend the bulk of their tokens on a discarded reasoning trace, driving up latency until requests exceed request_timeout. Set to None to omit the field entirely and leave the provider’s default behaviour unchanged, e.g. for a non-reasoning model or a provider that rejects unknown extra_body keys.

hybrid_initial_population: bool = False

Whether to include the LLM test cases in the initial population (pre-search seeding). Empirically, seeding spends token budget up front and delays the search without improving final coverage; the cost-coverage optimum is to leave this off and rely on call_llm_on_stall_detection (stagnation-triggered querying) instead.

llm_test_case_percentage: float = 0.5

The percentage of LLM test cases in on the initial population. The value must be from [0.0, 1.0].

llm_url: str = ''

Base URL for the LLM API endpoint (leave empty for default OpenAI).

max_context_chars: int = 64000

Maximum number of characters of module source code sent to the LLM. Coverage gains saturate beyond this budget while token consumption keeps growing, so the source is truncated to this length. Set to <= 0 to disable truncation.

max_llm_interventions: int = -1

The maximum number of allowed stall-triggered LLM interventions per run. Defaults to -1 (unlimited, bounded only by the search budget). This is safe because interventions are naturally rate-limited by the stall-detection window and the late-budget guard, so they fire at most once per detected plateau. Set to a non-negative value to cap the number of interventions per run instead.

max_plateau_len: int = 25

The number of iterations without a coverage change before soliciting the LLM. Only used as a fallback when stall_detection_window_seconds is <= 0; the time-based window is preferred because iteration duration varies widely.

max_request_time: float = 300.0

Total wall-clock budget (in seconds) for a single logical LLM request, covering every retry attempt and the backoff waits between them. A retry is only started when the budget can still accommodate it. Set to <= 0 to disable the bound.

max_retries: int = 8

The maximum number of retries for LLM requests.

min_remaining_budget_for_llm: int = 45

suppress stall-triggered LLM queries when fewer than this many seconds of search time remain, to avoid spending tokens on a query that cannot return and be integrated in time. Only applies when a maximum search time is configured; set to <= 0 to disable the guard.

Type:

Late-budget guard

model_name: str = 'gpt-4o-mini'

The OpenAI Model to use for completions.

request_timeout: float | None = 30.0

The timeout for LLM requests in seconds (None disables the timeout). A 30s timeout keeps query latency low during active evolutionary search. Callers that issue large batch requests outside the search (notably LLM refinement, see LLMRefinementConfiguration.request_timeout) override this per request.

stall_detection_window_seconds: int = 30

Wall-clock seconds without a coverage gain before the search is considered stalled and the LLM is queried (only used when call_llm_on_stall_detection is enabled). Roughly 10% of a standard 300s budget. Set to <= 0 to fall back to the iteration-count based max_plateau_len heuristic instead.

class pynguin.configuration.LLMRefinementConfiguration(enabled=False, max_repair_iterations=2, max_tests=None, request_timeout=180.0, save_original=True, save_refined=True, enable_dependency_context=False, enable_usage_examples=False, max_dependencies=10, max_usage_examples=3, enable_mutation_strengthening=False, max_mutation_iterations=3, refinement_granularity=RefinementGranularity.COMBINED)

Configuration for LLM-based test refinement.

This enables post-processing of generated tests using a large language model to improve readability and fix failing tests. The model name and API key are taken from LLMConfiguration (large_language_model.model_name and large_language_model.api_key) so that all LLM features share a single provider setup.

enable_dependency_context: bool = False

Enable retrieving dependent class/method signatures to include in prompt context.

enable_mutation_strengthening: bool = False

Enable mutation-driven assertion strengthening.

enable_usage_examples: bool = False

Enable retrieving usage examples from the codebase to include in prompt context.

enabled: bool = False

Enable LLM-based test refinement.

max_dependencies: int = 10

Maximum number of dependencies to collect context for.

max_mutation_iterations: int = 3

Maximum iterations for mutation-driven assertion strengthening.

max_repair_iterations: int = 2

Maximum number of iterations to attempt test repair. Capping the multi-turn syntax/type repair loop at 2 trades a tiny amount of test salvage for markedly lower latency and token usage.

max_tests: int | None = None

Maximum number of tests to refine (None = all tests).

max_usage_examples: int = 3

Maximum number of usage examples to collect context for.

refinement_granularity: RefinementGranularity = 'combined'

Whether readability refinement and semantic-assertion generation are done with a single combined module-level prompt (combined, default; most token-efficient, highest-variance), two separate module-level prompts (module_separate), or one prompt per test (per_test; previous behavior, most robust). Repair and mutation-strengthening are always per-test regardless of this setting. Any truncated or unparseable module-level response falls back automatically to the per-test path.

request_timeout: float | None = 180.0

Per-attempt timeout (in seconds) for refinement LLM requests. Refinement is batch post-processing whose module-level prompt carries the whole generated suite, so it needs more time than the search-calibrated large_language_model.request_timeout. None falls back to that value.

save_original: bool = True

Save the original unrefined test file.

save_refined: bool = True

Save the refined test file with _refined suffix.

class pynguin.configuration.LocalSearchConfiguration(local_search=True, local_search_same_datatype=True, local_search_different_datatype=False, local_search_llm=False, local_search_primitives=True, local_search_collections=False, local_search_complex_objects=False, local_search_probability=0.02, local_search_time=5000, ls_int_delta_increasing_factor=2, ls_string_random_mutation_count=10, ls_random_parametrized_statement_call_count=10, ls_max_different_type_mutations=10, ls_different_type_primitive_probability=0.3, ls_different_type_collection_probability=0.3, ls_dict_max_insertions=10, ls_llm_whole_module=False)

Configurations for local search.

local_search: bool = True

Defines if local search is generally enabled or not.

local_search_collections: bool = False

Toggles if local search for collection types is enabled or not

local_search_complex_objects: bool = False

Toggles if local search for complex object types is enabled or not

local_search_different_datatype: bool = False

Defines if local search with different datatypes is enabled or not

local_search_llm: bool = False

Defines if local search with llm is enabled or not

local_search_primitives: bool = True

Toggles if local search for primitive types is enabled or not

local_search_probability: float = 0.02

Probability of starting local search on the specific candidate

local_search_same_datatype: bool = True

Defines if local search within same datatype is enabled or not

local_search_time: int = 5000

The budget which each local search iteration has in ms.

ls_dict_max_insertions: int = 10

Max number of unsuccessful insertions of entries.

ls_different_type_collection_probability: float = 0.3

Weight of generating collection statements when changing the statement type.

ls_different_type_primitive_probability: float = 0.3

Weight of generating primitive statements when changing the statement type.

ls_int_delta_increasing_factor: int = 2

The factor which defines how much the delta should increase for each iteration for integer local search.

ls_llm_whole_module: bool = False

Whether to provide the whole module or selected functions/methods to the LLM for context when performing llm local search.

ls_max_different_type_mutations: int = 10

The max number of searches for different datatypes

ls_random_parametrized_statement_call_count: int = 10

The number of mutations made to a parametrized statement call to determine if it’s worth to further do local search

ls_string_random_mutation_count: int = 10

The number of mutations made to the string to determine if it’s worth to completely mutate the string.

class pynguin.configuration.MIOConfiguration(initial_config=<factory>, focused_config=<factory>, exploitation_starts_at_percent=0.5)

Configuration that is specific to the MIO approach.

exploitation_starts_at_percent: float = 0.5

Percentage ]0,1] of search budget after which exploitation is activated, i.e., switching to focused phase.

focused_config: MIOPhaseConfiguration

Configuration to use in focused phase

initial_config: MIOPhaseConfiguration

Configuration to use before focused phase.

class pynguin.configuration.MIOPhaseConfiguration(number_of_tests_per_target, random_test_or_from_archive_probability, number_of_mutations)

Configuration for a phase of MIO.

number_of_mutations: int

Number of mutations allowed to be done on the same individual before sampling a new one.

number_of_tests_per_target: int

Number of test cases for each target goal to keep in an archive.

random_test_or_from_archive_probability: float

Probability [0,1] of sampling a new test at random or choose an existing one in an archive.

class pynguin.configuration.Minimization(test_case_minimization_strategy=MinimizationStrategy.CASE, test_case_minimization_direction=MinimizationDirection.BACKWARD)

Configuration for test case minimization.

test_case_minimization_direction: MinimizationDirection = 'BACKWARD'

Direction to apply for minimizing test cases.

test_case_minimization_strategy: MinimizationStrategy = 'CASE'

Strategy to apply for minimizing test cases to remove redundant statements while preserving fitness.

class pynguin.configuration.MinimizationDirection(value)

Directions for test case minimization.

Either start with the first statement and try removing it and all forward dependent statements, or start with the last statement and try removing it and all backward dependent statements.

BACKWARD = 'BACKWARD'

Apply backward minimization, which removes statements from the end to the beginning.

FORWARD = 'FORWARD'

Apply forward minimization, which removes statements from the beginning to the end.

class pynguin.configuration.MinimizationStrategy(value)

Different strategies for minimizing test cases.

CASE = 'CASE'

Apply minimization at the test case level.

COMBINED = 'COMBINED'

Apply minimization at the test case level but check against all test cases in the suite.

NONE = 'NONE'

Do not apply any minimization.

SUITE = 'SUITE'

Apply minimization at the test suite level, i.e., remove entire test cases that do not contribute to the coverage of the module under test.

class pynguin.configuration.MutationStrategy(value)

Different strategies for creating mutants.

Only respected when using the MUTATION_ANALYSIS approach for assertion generation.

BETWEEN_OPERATORS = 'BETWEEN_OPERATORS'

Higher order mutation strategy BetweenOperators. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)

EACH_CHOICE = 'EACH_CHOICE'

Higher order mutation strategy EachChoice. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)

FIRST_ORDER_MUTANTS = 'FIRST_ORDER_MUTANTS'

Generate first order mutants.

FIRST_TO_LAST = 'FIRST_TO_LAST'

Higher order mutation strategy FirstToLast. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)

RANDOM = 'RANDOM'

Higher order mutation strategy Random. (cf. Mateo et al. Validating Second-Order Mutation at System Level. Article. IEEE Transactions on SE 39.4 2013)

class pynguin.configuration.PynguinMLConfiguration(ml_testing_enabled=False, constraints_path='', dtype_mapping_path='', constructor_function='', constructor_function_parameter='', max_ndim=4, max_shape_dim=4, ignore_constraints_probability=0.25)

Configurations for PynguinML, specifically for machine learning API testing.

constraints_path: str = ''

Directory path to YAML/JSON constraints files.

constructor_function: str = ''

Optional constructor function to build tensors (e.g., torch.tensor or tensorflow.convert_to_tensor). Must accept a np.ndarray as input. Note: Also set ‘constructor_function_parameter’ to specify which parameter receives the np.ndarray.

constructor_function_parameter: str = ''

Name of the parameter in the constructor function that receives the np.ndarray. Must be set when ‘constructor_function’ is specified.

dtype_mapping_path: str = ''

Path to a YAML/JSON file that maps library-specific datatypes to NumPy datatypes. Used by PynguinML for input generation. Specify only valid NumPy dtypes without the “np.” prefix. Example: “torch.int32” -> “int32”.

ignore_constraints_probability: float = 0.25

Probability of ignoring constraints and falling back to Pynguin’s default behavior when constructing statements.

max_ndim: int = 4

Maximum number of dimensions of the tensors. If set too big, it can cause memory errors due to large tensor sizes.

max_shape_dim: int = 4

Maximum size of a dimension of a shape. If set too big, it can cause memory errors due to large tensor sizes.

ml_testing_enabled: bool = False

Enables ML-based test generation using API constraints. When set to True, PynguinML will use constraints and other ML-specific behaviours during input generation.

Note: ‘constraints_path’ must also be set to locate the constraints files.

class pynguin.configuration.RandomConfiguration(max_sequence_length=10, max_sequences_combined=10)

Configuration that is specific to the RANDOM approach.

max_sequence_length: int = 10

The maximum length of sequences that are generated, 0 means infinite.

max_sequences_combined: int = 10

The maximum number of combined sequences, 0 means infinite.

class pynguin.configuration.RefinementGranularity(value)

Granularity of the LLM readability & assertion refinement stages.

COMBINED = 'combined'

One module-level prompt that does both readability and assertion refinement (default; most token-efficient, highest output variance).

MODULE_SEPARATE = 'module_separate'

readability first, then semantic assertions.

Type:

Two module-level prompts

PER_TEST = 'per_test'

Per-test readability and assertion prompts (previous behavior, most robust).

class pynguin.configuration.SearchAlgorithmConfiguration(min_initial_tests=1, max_initial_tests=10, population=10, chromosome_length=48, chop_max_length=True, elite=1, crossover_rate=0.648, test_insertion_probability=0.1, test_delete_probability=0.3333333333333333, test_change_probability=0.3333333333333333, test_insert_probability=0.3333333333333333, statement_insertion_probability=0.5, random_perturbation=0.2, change_parameter_probability=0.1, change_statement_type_probability=0.05, tournament_size=4, rank_bias=1.68, selection=Selection.RANK_SELECTION, use_archive=False, filter_covered_targets_from_test_cluster=False, number_of_mutations=3)

General configuration for search algorithms.

change_parameter_probability: float = 0.1

Probability of replacing parameters when mutating a method or constructor statement in a test case. Expects values in [0,1]

change_statement_type_probability: float = 0.05

Probability, per mutated statement, of replacing it with a statement of a different generated type (keeping its bound variable) instead of the regular value/call mutation. Expects values in [0,1].

chop_max_length: bool = True

Chop statements after exception if length has reached maximum

chromosome_length: int = 48

Maximum length of chromosomes during search

crossover_rate: float = 0.648

Probability of crossover

elite: int = 1

Elite size for search algorithm

filter_covered_targets_from_test_cluster: bool = False

Focus search by filtering out elements from the test cluster when they are fully covered.

max_initial_tests: int = 10

Maximum number of tests in initial test suites

min_initial_tests: int = 1

Minimum number of tests in initial test suites

number_of_mutations: int = 3

Number of mutations that should be applied in one breeding step.

population: int = 10

Population size of genetic algorithm

random_perturbation: float = 0.2

Probability to replace a primitive with a random new value rather than adding a delta.

rank_bias: float = 1.68

Bias for better individuals in rank selection

selection: Selection = 'RANK_SELECTION'

The selection operator for genetic algorithms.

statement_insertion_probability: float = 0.5

Initial probability of inserting a new statement in a test case

test_change_probability: float = 0.3333333333333333

Probability of changing statements during mutation

test_delete_probability: float = 0.3333333333333333

Probability of deleting statements during mutation

test_insert_probability: float = 0.3333333333333333

Probability of inserting new statements during mutation

test_insertion_probability: float = 0.1

Initial probability of inserting a new test in a test suite

tournament_size: int = 4

Number of individuals for tournament selection.

use_archive: bool = False

Some algorithms can be enhanced with an optional archive, e.g. Whole Suite -> Whole Suite + Archive. Use this option to enable the usage of an archive. Algorithms that always use an archive are not affected by this option.

class pynguin.configuration.SeedingConfiguration(seed=1788439421245978631, constant_seeding=True, initial_population_seeding=False, initial_population_data='', seeded_testcases_reuse_probability=0.9, initial_population_mutations=0, dynamic_constant_seeding=True, seeded_primitives_reuse_probability=0.2, seeded_dynamic_values_reuse_probability=0.6, seed_from_archive=False, seed_from_archive_probability=0.2, seed_from_archive_mutations=3, max_dynamic_length=1000, max_dynamic_pool_size=50)

Configuration related to seeding.

constant_seeding: bool = True

Should the generator use a static constant seeding technique to improve constant generation?

dynamic_constant_seeding: bool = True

Enables seeding of constants at runtime.

initial_population_data: str = ''

The path to the file with the pre-existing tests. The path has to include the file itself.

initial_population_mutations: int = 0

Number of how often the testcases collected by initial population seeding should be mutated to promote diversity

initial_population_seeding: bool = False

Should the generator use previously existing testcases to seed the initial population?

max_dynamic_length: int = 1000

Maximum length of strings/bytes that should be stored in the dynamic constant pool.

max_dynamic_pool_size: int = 50

Maximum number of constants of the same type that should be stored in the dynamic constant pool.

seed: int = 1788439421245978631

A predefined seed value for the random number generator that is used.

seed_from_archive: bool = False

When sampling new test cases reuse some from the archive, if one is used.

seed_from_archive_mutations: int = 3

Number of mutations applied when sampling from the archive.

seed_from_archive_probability: float = 0.2

Instead of creating a new test case, reuse a covering solution from the archive, iff an archive is used.

seeded_dynamic_values_reuse_probability: float = 0.6

Probability of using dynamically seeded values when a primitive seeded value will be used.

seeded_primitives_reuse_probability: float = 0.2

Probability for using seeded primitive values instead of randomly generated ones.

seeded_testcases_reuse_probability: float = 0.9

Probability of using seeded testcases when initial population seeding is enabled.

class pynguin.configuration.Selection(value)

Different selection algorithms to select from.

RANDOM_SELECTION = 'RANDOM_SELECTION'

Random selection.

RANK_SELECTION = 'RANK_SELECTION'

Rank selection.

TOURNAMENT_SELECTION = 'TOURNAMENT_SELECTION'

Tournament selection. Use tournament_size to set size.

class pynguin.configuration.StatisticsBackend(value)

The different available statistics backends to write statistics.

CONSOLE = 'CONSOLE'

Write statistics to the standard out.

CSV = 'CSV'

Write statistics to a CSV file.

NONE = 'NONE'

Do not write any statistics.

class pynguin.configuration.StatisticsOutputConfiguration(report_dir='pynguin-report', statistics_backend=StatisticsBackend.CSV, timeline_interval=1000000000, timeline_interpolation=True, coverage_metrics=<factory>, output_variables=<factory>, configuration_id='', run_id='', project_name='', create_coverage_report=False, type_guess_top_n=10, store_test_before_execution=False)

Configuration related to output.

configuration_id: str = ''

Label that identifies the used configuration of Pynguin. This is only done when running experiments.

coverage_metrics: list[CoverageMetric]

List of coverage metrics that are optimised during the search

create_coverage_report: bool = False

Create a coverage report for the tested module. This can be helpful to find hard to cover parts because Pynguin measures coverage on bytecode level which might yield different results when compared with other tools, e.g., Coverage.py.

output_variables: list[RuntimeVariable]

List of variables to output to the statistics backend.

project_name: str = ''

Label that identifies the project name of Pynguin. This is useful when running experiments.

report_dir: str = 'pynguin-report'

Directory in which to put HTML and CSV reports

run_id: str = ''

Id of the cluster run. Useful for finding the log entries that belong to a certain result.

statistics_backend: StatisticsBackend = 'CSV'

Which backend to use to collect data

store_test_before_execution: bool = False

When enabled, each test case is written to a file before execution and removed after successful completion. If a test causes a crash, the file remains. Use with caution: Converting pynguin internal representation to a pytest test case and writing it to a file is expensive and leads to lower iterations/coverage.

timeline_interpolation: bool = True

Interpolate timeline values

timeline_interval: int = 1000000000

Time interval in nano-seconds for timeline statistics, i.e., we select a data point after each interval. This can be interpolated, if there is no exact value stored at the time-step of the interval, see timeline_interpolation. The default value is every 1.00s.

type_guess_top_n: int = 10

When exporting type guesses for parameters, how many guesses per parameter should be exported? Expects positive integers.

class pynguin.configuration.StoppingConfiguration(maximum_search_time=-1, maximum_test_executions=-1, maximum_statement_executions=-1, maximum_slicing_time=600, maximum_iterations=-1, maximum_test_execution_timeout=5, maximum_coverage=100, maximum_coverage_plateau=-1, minimum_coverage=100, minimum_plateau_iterations=-1, maximum_memory=3000, test_execution_time_per_statement=1)

Configuration related to when Pynguin should stop.

Note that these are mostly soft-limits rather than hard limits, because the search algorithms only check the condition at the start of each algorithm iteration.

maximum_coverage: int = 100

The maximum percentage of coverage after which the generation shall stop.

maximum_coverage_plateau: int = -1

Maximum number of algorithm iterations without coverage change before the algorithms stops.

maximum_iterations: int = -1

Maximum iterations

maximum_memory: int = 3000

Maximum memory usage in MB after which the generation shall stop. Expects values in MB greater than 0 of -1 to disable the check.

maximum_search_time: int = -1

Time (in seconds) that can be used for generating tests.

maximum_slicing_time: int = 600

Time budget (in seconds) that can be used for slicing.

maximum_statement_executions: int = -1

Maximum number of test cases to be executed.

maximum_test_execution_timeout: int = 5

The maximum time (in seconds) after which a test case times out.

maximum_test_executions: int = -1

Maximum number of test cases to be executed.

minimum_coverage: int = 100

Minimum coverage for the plateau-based stopping condition. Expects values larger than 0 but less than 100 to activate the stopping condition; also requires the setting of minimum_plateau_iterations.

minimum_plateau_iterations: int = -1

Minimum iterations without a coverage change to stop early. Expects values larger than 0; also requires the setting of minimum_coverage.

test_execution_time_per_statement: int = 1

The time (in seconds) per statement that a test is allowed to run (up to maximum_test_execution_timeout).

class pynguin.configuration.StringStatementConfiguration(random_string_weight=0.3, faker_string_weight=0.3, fandango_string_weight=0.4, fandango_faker_string_weight=0.0, token_assembly_probability=0.2, max_assembled_tokens=4)

Configuration related to string statements.

faker_string_weight: float = 0.3

Probability use the Faker generator for string statement generation.

fandango_faker_string_weight: float = 0.0

Probability use the Fandango + Faker generator for string statement generation.

fandango_string_weight: float = 0.4

Probability use the Fandango generator for string statement generation.

max_assembled_tokens: int = 4

Upper bound on the number of seeded tokens concatenated into one string.

random_string_weight: float = 0.3

Probability use the random generator for string statement generation.

token_assembly_probability: float = 0.2

Probability to build a string by concatenating several seeded string constants.

Modules whose branches depend on token-structured inputs ("two thousand", "2h32m", "P1Y2M10D") define their vocabulary as string literals, so the tokens end up in the constant pool but are never combined. Assembling them lets the search reach those branches. Only applies when at least two distinct string constants were seeded.

class pynguin.configuration.SubtypeInferenceStrategy(value)

The different available type-inference strategies.

NONE = 'NONE'

Do not infer subtypes.

STRING = 'STRING'

Infer subtypes for strings.

class pynguin.configuration.TestCaseOutputConfiguration(output_path, crash_path='', export_strategy=ExportStrategy.PY_TEST, max_length_test_case=2500, assertion_generation=AssertionGenerator.MUTATION_ANALYSIS, allow_stale_assertions=False, filter_assertions_in_subprocess=True, mutation_strategy=MutationStrategy.FIRST_ORDER_MUTANTS, mutation_order=1, maximum_mutation_time=-1, maximum_mutants=-1, post_process=True, minimization=<factory>, float_precision=0.01, format_with_black=True, no_xfail=False, assertion_minimization=True)

Configuration related to test case output.

allow_stale_assertions: bool = False

Allow assertion on things that did not change between statement executions.

assertion_generation: AssertionGenerator = 'MUTATION_ANALYSIS'

The generator that shall be used for assertion generation.

assertion_minimization: bool = True

If enabled, remove redundant assertions after mutation analysis, keeping a minimal subset that preserves the set of killed mutants (greedy set cover).

crash_path: str = ''

Path to an output folder for the generated test cases that caused a crash. Only works when running in a subprocess. If set to “”, the are stored in the output_path.

export_strategy: ExportStrategy = 'PY_TEST'

The export strategy determines for which test-runner system the generated tests should fit.

filter_assertions_in_subprocess: bool = True

Run the assertion-filtering executions in a fresh subprocess. Filtering in a separate process exercises per-process nondeterminism (object id(), identity hashing, hash() of str/bytes, once-computed timestamps) that an in-process re-execution cannot observe, so assertions hard-coding such values are dropped instead of surviving to fail in a downstream pytest run.

float_precision: float = 0.01

Precision to use in float comparisons and assertions

format_with_black: bool = True

Format the generated test cases using black.

max_length_test_case: int = 2500

The maximum number of statement in as test case (normal + assertion statements)

maximum_mutants: int = -1

Maximum number of mutants checked during assertion generation. If the module yields more mutants, a seeded random sample of this size is used (-1 = unlimited).

maximum_mutation_time: int = -1

Maximum wall-clock time in seconds for executing tests against mutants during assertion generation (-1 = unlimited). When the budget is exceeded the current mutant finishes, then no further mutants are checked.

minimization: Minimization

Strategy to apply for minimizing test cases.

mutation_order: int = 1

The order of the generated higher order mutants in the mutation analysis assertion generation method.

mutation_strategy: MutationStrategy = 'FIRST_ORDER_MUTANTS'

The strategy that shall be used for creating mutants in the mutation analysis assertion generation method.

no_xfail: bool = False

If True, unexpected exceptions will be wrapped with pytest.raises() instead of marking the test with @pytest.mark.xfail(strict=True). Use this for better mutation scores.

output_path: str

Path to an output folder for the generated test cases.

post_process: bool = True

Should the results be post processed? For example, truncate test cases after statements that raise an exception.

class pynguin.configuration.TestCreationConfiguration(max_recursion=10, max_delta=20, max_int=2048, string_length=20, bytes_length=20, collection_size=5, collection_reference_probability=0.5, primitive_reuse_probability=0.5, object_reuse_probability=0.9, none_weight=1, any_weight=5, original_type_weight=5, type_tracing_weight=10, type_tracing_kept_guesses=2, wrap_var_param_type_probability=0.7, negate_type=0.1, skip_optional_parameter_probability=0.7, callable_argument_probability=0.25, callable_invocation_probability=0.25, max_attempts=1000, insertion_uut=0.5, max_size=100, use_random_object_for_call=0.1, generate_field_statements=False)

Configuration related to test creation.

any_weight: float = 5

Weight to use Any as parameter type during test generation. Expects values > 0.

bytes_length: int = 20

Maximum length of randomly generated bytes

callable_argument_probability: float = 0.25

Probability to satisfy a parameter of unknown (Any) type with a callable value – a cluster function, a class, a builtin such as str, or a synthesized lambda – instead of a literal. Parameters that are explicitly annotated as Callable always receive a callable value, independent of this setting. Expects values in [0,1].

callable_invocation_probability: float = 0.25

Probability to follow a statement whose value may be a callable (an unannotated or Callable-typed return value) with a statement that invokes it. This is what reaches the body of a closure returned by a higher-order function. Expects values in [0,1].

collection_reference_probability: float = 0.5

Probability that a collection element slot uses a reference to an existing in-scope variable instead of a fresh literal, when a reference pool is available. Expects values in [0,1].

collection_size: int = 5

Maximum length of randomly generated collections

generate_field_statements: bool = False

Whether to discover public fields/properties of analysed classes and emit attribute-access statements (var_1 = var_0.field) for them. Off by default because it is new behaviour relative to the class-based representation, where fields only ever entered generation through the LLM subsystem.

insertion_uut: float = 0.5

Score for selection of insertion of UUT calls

max_attempts: int = 1000

Number of attempts when generating an object before giving up

max_delta: int = 20

Maximum size of delta for numbers during mutation

max_int: int = 2048

Maximum size of randomly generated integers (minimum range = -1 * max)

max_recursion: int = 10

Recursion depth when trying to create objects in a test case.

max_size: int = 100

Maximum number of test cases in a test suite

negate_type: float = 0.1

When inferring a type from proxies, it may also be desirable to negate the chosen type, e.g., such that an instance check or a getattr() evaluate to False. Expects values in [0,1]

none_weight: float = 1

Weight to use None as parameter type during test generation. Expects values > 0.

object_reuse_probability: float = 0.9

Probability to reuse an existing object in a test case, if available. Expects values in [0,1]

original_type_weight: float = 5

Weight to use the originally annotated type as parameter type during test generation. Expects values > 0.

primitive_reuse_probability: float = 0.5

Probability to reuse an existing primitive in a test case, if available. Expects values in [0,1]

skip_optional_parameter_probability: float = 0.7

Probability to skip an optional parameter, i.e., do not fill such a parameter.

string_length: int = 20

Maximum length of randomly generated strings

type_tracing_kept_guesses: int = 2

Amount of kept recently guessed types per parameter, when type tracing is used.

type_tracing_weight: float = 10

Weight to use the type guessed from type tracing as parameter type during test generation. Expects values > 0.

use_random_object_for_call: float = 0.1

When adding or modifying a call on an object, use a random modifier instead of only modifiers for that type. Expects values in [0, 1].

wrap_var_param_type_probability: float = 0.7

Probability to wrap the type required for a *arg or **kwargs parameter in a list or dict, respectively. Expects values in [0,1]

class pynguin.configuration.ToCoverConfiguration(only_cover=<factory>, no_cover=<factory>, enable_inline_pynguin_no_cover=True, enable_inline_pragma_no_cover=True)

Configuration of which code elements are included or excluded as coverage goals.

Pynguin instruments only the respective parts of the SUT module for coverage measurement. All configuration options are resolved as SUT line numbers which must not overlap (‘only’ and ‘no’ at the same time).

enable_inline_pragma_no_cover: bool = True

Enable inline pragma: no cover.

enable_inline_pynguin_no_cover: bool = True

Enable inline pynguin: no cover.

no_cover: list[str]

The list of space-separated qualified function, method or class names to not cover. Automatically include the methods of the ignore_methods argument.

only_cover: list[str]

The list of space-separated qualified function, method or class names to only cover.

class pynguin.configuration.TypeInferenceConfiguration(type_inference_strategy=TypeInferenceStrategy.TYPE_HINTS, type_tracing=0.0, subtype_inference=SubtypeInferenceStrategy.NONE, type_tracing_subtype_weight=0.3, type_tracing_argument_type_weight=0.5, type_tracing_attribute_weight=0.2, typeevalpy_json_path='')

Configuration related to type inference.

subtype_inference: SubtypeInferenceStrategy = 'NONE'

The strategy for subtype-inference that shall be used.

type_inference_strategy: TypeInferenceStrategy = 'TYPE_HINTS'

The strategy for type-inference that shall be used

type_tracing: bool | float = 0.0

Probability to trace usage of parameters with unknown types to improve type guesses during test execution. Type tracing requires a separate second. The value should be a float in [0,1]. Boolean is kept for backwards compatibility as Python internally converts True to 1.0 and False to 0.0 anyways.

type_tracing_argument_type_weight: float = 0.5

Weight for selecting the argument type inference strategy for type selection during type tracing.

type_tracing_attribute_weight: float = 0.2

Weight for selecting the attribute table inference strategy for type selection during type tracing.

type_tracing_subtype_weight: float = 0.3

Weight for selecting the subtype inference strategy for type selection during type tracing.

typeevalpy_json_path: str = ''

Path to a TypeEvalPy JSON file containing type information to use during test generation. If provided, the file will be parsed and the type information will be used to improve type inference.

class pynguin.configuration.TypeInferenceStrategy(value)

The different available type-inference strategies.

LLM = 'LLM'

Use an LLM to infer types for the module under test.

LLM_WITH_SUBTYPES = 'LLM_WITH_SUBTYPES'

Use an LLM to infer types and subtypes for the module under test.

NONE = 'NONE'

Ignore any type information given in the module under test.

TYPEEVALPY = 'TYPEEVALPY'

Use TypeEvalPy data only for type inference.

TYPE_HINTS = 'TYPE_HINTS'

Use type information from type hints in the module under test.

pynguin.generator

Pynguin is an automated unit test generation framework for Python.

The framework generates unit tests for a given Python module. For this it supports various approaches, such as a random approach, similar to Randoop or a whole-suite approach, based on a genetic algorithm, as implemented in EvoSuite. The framework allows to export test suites in various styles, i.e., using the unittest library from the Python standard library or tests in the style used by the PyTest framework.

Pynguin is supposed to be used as a standalone command-line application but it can also be used as a library by instantiating this class directly.

class pynguin.generator.ReturnCode(value)

Return codes for Pynguin to signal result.

FINAL_METRICS_TRACKING_FAILED = 3

Symbolises that the final metrics tracking failed.

NO_TESTS_GENERATED = 2

Symbolises that no test could be generated.

OK = 0

Symbolises that the execution ended as expected.

SETUP_FAILED = 1

Symbolises that the execution failed in the setup phase.

pynguin.generator.run_pynguin()

Run the test generation.

The result of the test generation is indicated by the resulting ReturnCode.

Return type:

ReturnCode

Returns:

See ReturnCode.

Raises:

ConfigurationException – In case the configuration is illegal

pynguin.generator.set_configuration(configuration)

Initialises the test generator with the given configuration.

Parameters:

configuration (Configuration) – The configuration to use.

Return type:

None

pynguin.analyses.module

Provides analyses for the subject module, based on the module and its AST.

class pynguin.analyses.module.CallableData(accessible, tree, description, cyclomatic_complexity)

Provides all information on callables.

While the accessible is available for every callable, the other fields are only filled for methods that are available in (Python) source code because their information is retrieved from the abstract syntax tree.

accessible

the accessible object itself

tree

the AST of the callable, if any

description

the function description of the callable, if any

cyclomatic_complexity

the McCabe cyclomatic complexity of the callable, if any

class pynguin.analyses.module.FilteredModuleTestCluster(delegate, archive, subject_properties, targets)

A test cluster wrapping another test cluster.

Delegates most methods to the wrapped delegate. This cluster filters out accessible objects under test that are already fully covered, in order to focus the search on areas that are not yet fully covered.

property accessible_objects_under_test: OrderedSet[GenericAccessibleObject]

Provides all accessible objects under test.

add_accessible_object_under_test(objc, data)

Add accessible object to the objects under test.

Parameters:

• objc (GenericAccessibleObject) – The accessible object

• data (CallableData) – The function-description data

Return type:

None

add_generator(generator)

Add the given accessible as a generator.

Parameters:

generator (GenericAccessibleObject) – The accessible object

Return type:

None

add_ml_data(obj, data)

Provides ML data for a accessible.

Return type:

None

add_modifier(typ, obj)

Add a modifier.

A modifier is something that can be used to modify the given type, for example, a method.

Parameters:

• typ (TypeInfo) – The type that can be modified

• obj (GenericAccessibleObject) – The accessible that can modify

Return type:

None

property function_data_for_accessibles: dict[GenericAccessibleObject, CallableData]

Provides all function data for all accessibles.

property generators: dict[ProperType, OrderedSet[GenericAccessibleObject]]

Provides all available generators.

get_all_generatable_types()

Provides all types that can be generated.

This includes primitives and collections.

Returns:

DAR202

Return type:

A list of all types that can be generated # noqa

get_generators_for(typ)

Retrieve all known generators for the given type.

Parameters:

typ (ProperType) – The type we want to have the generators for

Return type:

OrderedSet[GenericAccessibleObject]

Returns:

The set of all generators for that type.

get_ml_data_for(obj)

Provides ML data for a accessible.

Return type:

MLCallableData | None

get_modifiers_for(typ)

Get all known modifiers for a type.

Parameters:

typ (ProperType) – The type

Returns:

DAR202

Return type:

The set of all accessibles that can modify the type # noqa

get_random_accessible()

Provides a random accessible of the unit under test.

Returns:

DAR202

Return type:

A random accessible, or None if there is none # noqa

get_random_call_for(typ)

Get a random modifier for the given type.

Parameters:

typ (ProperType) – The type

Returns:

DAR202

Return type:

A random modifier for that type # noqa

Raises:

ConstructionFailedException – if no modifiers for the type exist# noqa: DAR402

property linenos: int

Provide the number of source code lines.

log_cluster_statistics()

Log the signatures of all seen callables.

Return type:

None

property modifiers: dict[TypeInfo, OrderedSet[GenericAccessibleObject]]

Provides all available modifiers.

num_accessible_objects_under_test()

Provide the number of accessible objects under test.

Useful to check whether there is even something to test.

Return type:

int

on_target_covered(target)

A callback function to get informed by an archive when a target is covered.

Parameters:

target (TestCaseFitnessFunction) – The newly covered target

Return type:

None

select_concrete_type(typ)

Select a concrete type from the given type.

This is required, for example, when handling union types. Currently, only unary types, Any, and Union are handled.

Parameters:

typ (ProperType) – An optional type

Returns:

DAR202

Return type:

An optional type # noqa

track_statistics_values(tracking_fun)

Track statistics values from the test cluster and its items.

Parameters:

tracking_fun (Callable[[RuntimeVariable, Any], None]) – The tracking function as a callback.

Return type:

None

property type_system: TypeSystem

Provides the inheritance graph.

update_parameter_knowledge(accessible, param_name, knowledge)

Update the knowledge about the parameter of the given accessible.

Parameters:

• accessible (GenericCallableAccessibleObject) – the accessible that was observed.

• param_name (str) – the parameter name for which we have new information.

• knowledge (UsageTraceNode) – the new information.

Return type:

None

update_return_type(accessible, new_type)

Update the return for the given accessible to the new seen type.

Parameters:

• accessible (GenericCallableAccessibleObject) – the accessible that was observed

• new_type (ProperType) – the new return type

Return type:

None

class pynguin.analyses.module.MLCallableData(parameters, generation_order)

Provides ML-specific information on callables.

parameters

A dictionary of parameters, if any

generation_order

The generation order of the callable (can be empty)

class pynguin.analyses.module.ModuleTestCluster(linenos)

A test cluster for a module.

Contains all methods/constructors/functions and all required transitive dependencies.

property accessible_objects_under_test: OrderedSet[GenericAccessibleObject]

Provides all accessible objects under test.

add_accessible_object_under_test(objc, data)

Add accessible object to the objects under test.

Parameters:

• objc (GenericAccessibleObject) – The accessible object

• data (CallableData) – The function-description data

Return type:

None

add_generator(generator)

Add the given accessible as a generator.

Parameters:

generator (GenericAccessibleObject) – The accessible object

Return type:

None

add_ml_data(obj, data)

Provides ML data for a accessible.

Return type:

None

add_modifier(typ, obj)

Add a modifier.

A modifier is something that can be used to modify the given type, for example, a method.

Parameters:

• typ (TypeInfo) – The type that can be modified

• obj (GenericAccessibleObject) – The accessible that can modify

Return type:

None

property function_data_for_accessibles: dict[GenericAccessibleObject, CallableData]

Provides all function data for all accessibles.

property generators: dict[ProperType, OrderedSet[GenericAccessibleObject]]

Provides all available generators.

get_all_generatable_types()

Provides all types that can be generated.

This includes primitives and collections.

Returns:

DAR202

Return type:

A list of all types that can be generated # noqa

get_generators_for(typ)

Retrieve all known generators for the given type.

Parameters:

typ (ProperType) – The type we want to have the generators for

Return type:

OrderedSet[GenericAccessibleObject]

Returns:

The set of all generators for that type.

get_ml_data_for(obj)

Provides ML data for a accessible.

Return type:

MLCallableData | None

get_modifiers_for(typ)

Get all known modifiers for a type.

Parameters:

typ (ProperType) – The type

Returns:

DAR202

Return type:

The set of all accessibles that can modify the type # noqa

get_random_accessible()

Provides a random accessible of the unit under test.

Returns:

DAR202

Return type:

A random accessible, or None if there is none # noqa

get_random_call_for(typ)

Get a random modifier for the given type.

Parameters:

typ (ProperType) – The type

Returns:

DAR202

Return type:

A random modifier for that type # noqa

Raises:

ConstructionFailedException – if no modifiers for the type exist# noqa: DAR402

property linenos: int

Provide the number of source code lines.

log_cluster_statistics()

Log the signatures of all seen callables.

Return type:

None

property modifiers: dict[TypeInfo, OrderedSet[GenericAccessibleObject]]

Provides all available modifiers.

num_accessible_objects_under_test()

Provide the number of accessible objects under test.

Useful to check whether there is even something to test.

Return type:

int

select_concrete_type(typ)

Select a concrete type from the given type.

This is required, for example, when handling union types. Currently, only unary types, Any, and Union are handled.

Parameters:

typ (ProperType) – An optional type

Returns:

DAR202

Return type:

An optional type # noqa

track_statistics_values(tracking_fun)

Track statistics values from the test cluster and its items.

Parameters:

tracking_fun (Callable[[RuntimeVariable, Any], None]) – The tracking function as a callback.

Return type:

None

property type_system: TypeSystem

Provides the type system.

Returns:

The type system.

update_parameter_knowledge(accessible, param_name, knowledge)

Update the knowledge about the parameter of the given accessible.

Parameters:

• accessible (GenericCallableAccessibleObject) – the accessible that was observed.

• param_name (str) – the parameter name for which we have new information.

• knowledge (UsageTraceNode) – the new information.

Return type:

None

update_return_type(accessible, new_type)

Update the return for the given accessible to the new seen type.

Parameters:

• accessible (GenericCallableAccessibleObject) – the accessible that was observed

• new_type (ProperType) – the new return type

Return type:

None

class pynguin.analyses.module.SignatureInfo(annotated_parameter_types=<factory>, guessed_parameter_types=<factory>, partial_type_matches=<factory>, annotated_return_type=None, recorded_return_type=None)

Another utility class to group information per callable.

class pynguin.analyses.module.TestCluster

Interface for a test cluster.

abstract property accessible_objects_under_test: OrderedSet[GenericAccessibleObject]

Provides all accessible objects under test.

abstract add_accessible_object_under_test(objc, data)

Add accessible object to the objects under test.

Parameters:

• objc (GenericAccessibleObject) – The accessible object

• data (CallableData) – The function-description data

Return type:

None

abstract add_generator(generator)

Add the given accessible as a generator.

Parameters:

generator (GenericAccessibleObject) – The accessible object

Return type:

None

abstract add_ml_data(obj, data)

Provides ML data for a accessible.

Return type:

None

abstract add_modifier(typ, obj)

Add a modifier.

A modifier is something that can be used to modify the given type, for example, a method.

Parameters:

• typ (TypeInfo) – The type that can be modified

• obj (GenericAccessibleObject) – The accessible that can modify

Return type:

None

abstract property function_data_for_accessibles: dict[GenericAccessibleObject, CallableData]

Provides all function data for all accessibles.

abstract property generators: dict[ProperType, OrderedSet[GenericAccessibleObject]]

Provides all available generators.

abstract get_all_generatable_types()

Provides all types that can be generated.

This includes primitives and collections.

Returns:

DAR202

Return type:

A list of all types that can be generated # noqa

abstract get_generators_for(typ)

Retrieve all known generators for the given type.

Parameters:

typ (ProperType) – The type we want to have the generators for

Return type:

OrderedSet[GenericAccessibleObject]

Returns:

The set of all generators for that type.

abstract get_ml_data_for(generic_accessible)

Provides ML data for a accessible.

Return type:

MLCallableData | None

abstract get_modifiers_for(typ)

Get all known modifiers for a type.

Parameters:

typ (ProperType) – The type

Returns:

DAR202

Return type:

The set of all accessibles that can modify the type # noqa

abstract get_random_accessible()

Provides a random accessible of the unit under test.

Returns:

DAR202

Return type:

A random accessible, or None if there is none # noqa

abstract get_random_call_for(typ)

Get a random modifier for the given type.

Parameters:

typ (ProperType) – The type

Returns:

DAR202

Return type:

A random modifier for that type # noqa

Raises:

ConstructionFailedException – if no modifiers for the type exist# noqa: DAR402

abstract property linenos: int

Provide the number of source code lines.

abstract log_cluster_statistics()

Log the signatures of all seen callables.

Return type:

None

abstract property modifiers: dict[TypeInfo, OrderedSet[GenericAccessibleObject]]

Provides all available modifiers.

abstract num_accessible_objects_under_test()

Provide the number of accessible objects under test.

Useful to check whether there is even something to test.

Return type:

int

abstract select_concrete_type(typ)

Select a concrete type from the given type.

This is required, for example, when handling union types. Currently, only unary types, Any, and Union are handled.

Parameters:

typ (ProperType) – An optional type

Returns:

DAR202

Return type:

An optional type # noqa

abstract track_statistics_values(tracking_fun)

Track statistics values from the test cluster and its items.

Parameters:

tracking_fun (Callable[[RuntimeVariable, Any], None]) – The tracking function as a callback.

Return type:

None

abstract property type_system: TypeSystem

Provides the inheritance graph.

abstract update_parameter_knowledge(accessible, param_name, knowledge)

Update the knowledge about the parameter of the given accessible.

Parameters:

• accessible (GenericCallableAccessibleObject) – the accessible that was observed.

• param_name (str) – the parameter name for which we have new information.

• knowledge (UsageTraceNode) – the new information.

Return type:

None

abstract update_return_type(accessible, new_type)

Update the return for the given accessible to the new seen type.

Parameters:

• accessible (GenericCallableAccessibleObject) – the accessible that was observed

• new_type (ProperType) – the new return type

Return type:

None

class pynguin.analyses.module.TypeGuessingStats(number_of_constructors=0, signature_infos=<factory>)

Class to gather some type guessing related statistics.

pynguin.analyses.module.analyse_module(parsed_module, type_inference_strategy=TypeInferenceStrategy.TYPE_HINTS)

Analyses a module to build a test cluster.

Parameters:

• parsed_module (_ModuleParseResult) – The parsed module

• type_inference_strategy (TypeInferenceStrategy) – The type inference strategy to use.

Return type:

ModuleTestCluster

Returns:

A test cluster for the module

pynguin.analyses.module.collect_provider_metrics(typ_provider)

Collects metrics from the given type inference provider.

currently, this only works for LLM-based providers.

When using an LLM-based provider, collect the raw inferred parameter strings per callable and the annotated parameter strings as JSON and store in the LLMInferredSignatures runtime variable.

Other providers default all metrics to zero.

pynguin.analyses.module.generate_test_cluster(module_name, type_inference_strategy=TypeInferenceStrategy.TYPE_HINTS)

Generates a new test cluster from the given module.

Parameters:

• module_name (str) – The name of the root module

• type_inference_strategy (TypeInferenceStrategy) – Which type-inference strategy to use

Return type:

ModuleTestCluster

Returns:

A new test cluster for the given module

pynguin.analyses.module.get_type_provider(type_inference_strategy, module, type_system)

Get the initialised inference provider for the given strategy.

Parameters:

• type_inference_strategy (TypeInferenceStrategy) – The type inference strategy to use

• module (ModuleType) – The module to analyse (only needed for LLM-based inference)

• type_system (TypeSystem) – The type system to use

Return type:

InferenceProvider

Returns:

The type inference provider for the given strategy

pynguin.analyses.module.import_module(module_name)

Imports a module by name.

Unlike the built-in importlib.import_module(), this function also supports importing module aliases.

Parameters:

module_name (str) – The fully-qualified name of the module

Return type:

ModuleType

Returns:

The imported module

pynguin.analyses.module.parse_module(module_name)

Parses a module and extracts its module-type and AST.

If the source code is not available it is not possible to build an AST. In this case the respective field of the _ModuleParseResult will contain the value None. This is the case, for example, for modules written in native code, for example, in C.

Parameters:

module_name (str) – The fully-qualified name of the module

Return type:

_ModuleParseResult

Returns:

A tuple of the imported module type and its optional AST

pynguin.analyses.module.read_module_ast(module_path)

Reads the AST of the module and returns it along with its source code.

Parameters:

module_path (str) – The path of the module.

Raises:

• OSError – if the module file cannot be read.

• SyntaxError – if an error occurs during the creation of the AST.

Return type:

tuple[Module, str]

Returns:

A tuple containing the AST and the source code.

pynguin.analyses.typesystem

Provides analyses for a module’s type information.

class pynguin.analyses.typesystem.AnyType

The Any Type.

accept(visitor)

Accept a type visitor.

Parameters:

visitor (TypeVisitor[TypeVar(T)]) – the visitor

Return type:

TypeVar(T)

class pynguin.analyses.typesystem.CSVString

A string that represents comma-separated values.

Examples: “value1,value2,value3”, “itemA,itemB,itemC”.

class pynguin.analyses.typesystem.EmailString

A string that follows the general pattern of an email address.

Example: “name@domain.edu”.

class pynguin.analyses.typesystem.HexadecimalString

A string that represents a hexadecimal number.

Examples: “0x1A3F”, “1a3f”, “0XABCDEF”.

class pynguin.analyses.typesystem.IPv4String

A string that represents an IPv4 address.

Examples: “192.168.0.0”, “74.5.123.31”.

class pynguin.analyses.typesystem.IPv6String

A string that represents an IPv6 address.

Examples: “2001:0db8:85a3:0000:0000:8a2e:0370:7334”, “fe80::1ff:fe23:4567:890a”.

class pynguin.analyses.typesystem.ISOColorString

A string that represents a hexadecimal color code.

Examples: “#FFFFFF”, “#000000”, “#FF5733”.

class pynguin.analyses.typesystem.ISODateString

A string that represents a date in ISO 8601 format.

Examples: “2023-10-05”, “1999-12-31”.

class pynguin.analyses.typesystem.ISOTimeString

A string that represents a time in ISO 8601 format.

Examples: “14:30:00”, “23:59:59”.

class pynguin.analyses.typesystem.InferredSignature(signature, original_return_type, original_parameters, type_system, parameters_for_statistics=<factory>, return_type_for_statistics=AnyType())

Encapsulates the types inferred for a method.

get_parameter_types(signature_memo)

Get a possible type signature for the parameters.

This method may choose a random type signature, or return the original one or create one based on the observed knowledge.

Parameters:

signature_memo (dict[InferredSignature, dict[str, ProperType]]) – A memo that stores already chosen signatures, so that we don’t choose another signature in the same run. This is required for certain operations in the test factory to be consistent.

Return type:

dict[str, ProperType]

Returns:

A dict of chosen parameter types for each parameter.

log_stats_and_guess_signature(is_constructor, callable_full_name, stats)

Logs some statistics and creates a guessed signature.

Parameters annotated with Any could not be guessed.

Parameters:

• callable_full_name (str) – The full, unique name of the callable.

• is_constructor (bool) – does this signature to a constructor?

• stats (TypeGuessingStats) – stats object to log to.

Return type:

None

class pynguin.analyses.typesystem.Instance(typ, args=None)

An instance type of form C[T1, …, Tn].

C is a class. Args can be empty.

accept(visitor)

Accept a type visitor.

Parameters:

visitor (TypeVisitor[TypeVar(T)]) – the visitor

Return type:

TypeVar(T)

class pynguin.analyses.typesystem.NamedDefaultDict

A default dictionary that automatically creates nodes for keys.

Default dict which automatically creates a UsageTraceNode for each requested and non-existing key.

class pynguin.analyses.typesystem.NoneType

The None type.

accept(visitor)

Accept a type visitor.

Parameters:

visitor (TypeVisitor[TypeVar(T)]) – the visitor

Return type:

TypeVar(T)

class pynguin.analyses.typesystem.NumericString

A string that only contains numerical characters and an optional decimal point.

Examples: “123”, “123.45”, “0.001”

class pynguin.analyses.typesystem.PhoneNumberString

A string that represents a phone number.

Examples: “+1-800-555-1234”, “(123) 456-7890”.

class pynguin.analyses.typesystem.ProperType

Base class for all types. Might have to add another layer, like mypy’s Type?.

All subclasses of this class are immutable.

abstract accept(visitor)

Accept a type visitor.

Parameters:

visitor (TypeVisitor[TypeVar(T)]) – the visitor

Return type:

TypeVar(T)

class pynguin.analyses.typesystem.SHA256String

A string that represents a SHA-256 hash.

Examples: “e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855”.

class pynguin.analyses.typesystem.StringSubtype(regex)

A subtype of str determined by its regex.

TODO(lk): Add support to all visitors.

accept(visitor)

Accept a type visitor.

Parameters:

visitor (TypeVisitor[TypeVar(T)]) – the visitor

Return type:

TypeVar(T)

class pynguin.analyses.typesystem.TupleType(args, *, unknown_size=False)

Tuple type Tuple[T1, …, Tn].

Note that tuple is a special case and intentionally not Instance(TypeInfo(tuple)) because tuple is varargs generic.

accept(visitor)

Accept a type visitor.

Parameters:

visitor (TypeVisitor[TypeVar(T)]) – the visitor

Return type:

TypeVar(T)

class pynguin.analyses.typesystem.TypeInfo(raw_type)

A small wrapper around type, i.e., classes.

Corresponds 1:1 to a class.

static get_dunder_value_from_type(typ, name)

Get the dunder value with the given name from the given type.

If the given type has no dunder attribute with the given name, we fall back to using the type of the given typ (== a value in this case). This worked for the UnionType.

Parameters:

• typ (type) – The type from which to get the attribute.

• name (str) – The name of the dunder attribute to get.

Return type:

str

Returns:

The value of the dunder attribute.

static to_full_name(typ)

Get the full name of the given type.

While type has a __name__, __qualname__ and __module__ attribute, UnionType does not. This caused a crash which is resolved by special handling of UnionType.

Parameters:

typ – The type for which we want a full name.

Return type:

str

Returns:

The fully qualified name

class pynguin.analyses.typesystem.TypeReprVisitor

A simple visitor to create a repr from a proper type.

visit_any_type(left)

Visit the Any type.

Parameters:

left (AnyType) – the Any type

Return type:

str

Returns:

result of the visit

visit_instance(left)

Visit an instance.

Parameters:

left (Instance) – instance

Return type:

str

Returns:

result of the visit

visit_none_type(left)

Visit the None type.

Parameters:

left (NoneType) – the None type

Return type:

str

Returns:

result of the visit

visit_tuple_type(left)

Visit a tuple type.

Parameters:

left (TupleType) – tuple

Return type:

str

Returns:

result of the visit

visit_union_type(left)

Visit a union.

Parameters:

left (UnionType) – union

Return type:

str

Returns:

result of the visit

visit_unsupported_type(left)

Visit unsupported type.

Parameters:

left (Unsupported) – unsupported

Return type:

str

Returns:

result of the visit

class pynguin.analyses.typesystem.TypeStringVisitor

A simple visitor to convert a proper type to a string.

visit_any_type(left)

Visit the Any type.

Parameters:

left (AnyType) – the Any type

Return type:

str

Returns:

result of the visit

visit_instance(left)

Visit an instance.

Parameters:

left (Instance) – instance

Return type:

str

Returns:

result of the visit

visit_none_type(left)

Visit the None type.

Parameters:

left (NoneType) – the None type

Return type:

str

Returns:

result of the visit

visit_tuple_type(left)

Visit a tuple type.

Parameters:

left (TupleType) – tuple

Return type:

str

Returns:

result of the visit

visit_union_type(left)

Visit a union.

Parameters:

left (UnionType) – union

Return type:

str

Returns:

result of the visit

visit_unsupported_type(left)

Visit unsupported type.

Parameters:

left (Unsupported) – unsupported

Return type:

str

Returns:

result of the visit

class pynguin.analyses.typesystem.TypeSystem

Implements Pynguin’s internal type system.

Provides a simple inheritance graph relating various classes using their subclass relationships. Note that parents point to their children.

This is also the central system to store/handle type information.

add_subclass_edge(*, super_class, sub_class)

Add a subclass edge between two types.

Parameters:

• super_class (TypeInfo) – superclass

• sub_class (TypeInfo) – subclass

Return type:

None

convert_type_hint(hint, unsupported=AnyType())

Converts a type hint to a proper type.

Python’s builtin functionality makes handling types during runtime really hard, because 1) this is not intended to be used at runtime and 2) there are a lot of different notations, due to the constantly evolving type hint system. We also cannot easily use mypy’s type abstraction because it is 1) strongly encapsulated and not part of mypy’s public API and 2) is designed to be used for static type checking. This method tries to translate type hints into our own type abstraction in order to make handling types less painful.

This conversion is naive when compared to what sophisticated type checkers like mypy do, but it is hopefully sufficient for our purposes. This method only handles a very small subset of the types that we may encounter in the wild, but at least it allows use to better reason about types. This should be extended in the future to handle more cases.

Parameters:

• hint (Any) – The type hint

• unsupported (ProperType) – The type to use when encountering an unsupported type construct

Return type:

ProperType

Returns:

A proper type.

property dot: str

Create dot representation of this graph.

Returns:

A dot string.

enable_numeric_tower()

Enable the numeric tower on this type system.

find_by_attribute(attr)

Search for all types that have the given attribute.

Parameters:

attr (str) – the attribute to search for.

Return type:

OrderedSet[TypeInfo]

Returns:

All types (or supertypes thereof) who have the given attribute.

find_type_info(full_name)

Find typeinfo for the given name.

Parameters:

full_name (str) – The name to search for.

Return type:

TypeInfo | None

Returns:

Type info, if any.

get_all_types()

Provides a list of all known types.

Return type:

list[TypeInfo]

Returns:

A list of all known types.

get_shortest_path_length(start, end)

Get the shortest path length between two types.

I tried adding @functools.lru_cache to this function does not improve performance, because we cache subtype_distance anyway. I tried using a _shortest_path_length_cache[(start, end)] along with using shortest_path_length(graph, source), which returns all nodes reachable from start and its length. However, probably the overhead from caching and having many different starting nodes make this slower than just using shortest_path_length with a target node (which can also leverage the quicker bidirectional_shortest_path internally). I also tried to come up with an own variant of nx.single_shortest_path_length that would stop at the target node, thinking the overhead maybe comes form deeper traversal, but this did not achieve better performance either. Thus, I suggest we stick to the simple version.

Parameters:

• start (TypeInfo) – The start type

• end (TypeInfo) – The end type

Return type:

int | None

Returns:

The shortest path length between the two types or None if no path exists.

get_subclasses(klass)

Provides all descendants of the given type. Includes klass.

Parameters:

klass (TypeInfo) – The class whose subtypes we want to query.

Return type:

OrderedSet[TypeInfo]

Returns:

All subclasses including klass

get_superclasses(klass)

Provides all ancestors of the given class.

Parameters:

klass (TypeInfo) – The class whose supertypes we want to query.

Return type:

OrderedSet[TypeInfo]

Returns:

All superclasses including klass

get_type_outside_of(klasses)

Find a type that does not belong to the given types or any subclasses.

Parameters:

klasses (OrderedSet[TypeInfo]) – The classes to exclude

Return type:

OrderedSet[TypeInfo]

Returns:

A set of klasses that don’t belong the given ones.

infer_signature(method, type_hint_provider)

Infers the method signature using the given type hint provider.

Parameters:

• method (Callable) – The callable

• type_hint_provider (Callable[[Callable], dict]) – A method that provides type hints for the given method.

Return type:

InferredSignature

Returns:

The inference result

infer_type_info(method, type_inference_provider)

Infers the type information for a callable.

Parameters:

• method (Callable) – The callable we try to infer type information for

• type_inference_provider (InferenceProvider) – The provider for type inference

Return type:

InferredSignature

Returns:

The inference result

is_maybe_subtype(left, right)

Is ‘left’ maybe a subtype of ‘right’?

This is a more lenient check than is_subtype. Consider a function that returns tuple[str | int | bytes, str | int | bytes]. Strictly speaking, we cannot use such a value as an argument for a function that requires an argument of type tuple[int, int]. However, it may be possible that the returned value is tuple[int, int], in which case it does work. This check only differs from is_subtype in how it handles Unions. Instead of requiring every type to be a subtype, it is sufficient that one type of the Union is a subtype.

Parameters:

• left (ProperType) – The left type

• right (ProperType) – The right type

Return type:

bool

Returns:

True, if left may be a subtype of right.

is_subclass(left, right)

Is ‘left’ a subclass of ‘right’?

Parameters:

• left (TypeInfo) – left type info

• right (TypeInfo) – right type info

Return type:

bool

Returns:

True, if there is a subclassing path from left to right.

is_subtype(left, right)

Is ‘left’ a subtype of ‘right’?

This check is more than incomplete, but it takes into account that anything is a subtype of AnyType.

See https://peps.python.org/pep-0483/ and https://peps.python.org/pep-0484/ for more details

Parameters:

• left (ProperType) – The left type

• right (ProperType) – The right type

Return type:

bool

Returns:

True, if left is a subtype of right.

make_instance(typ)

Create an instance from the given type.

Parameters:

typ (TypeInfo) – The type info.

Return type:

Instance | TupleType | NoneType

Returns:

An instance or TupleType

push_attributes_down()

Pushes attributes down in hierarchy.

We don’t want to see attributes multiple times, e.g., in subclasses, so only the first class in the hierarchy which adds the attribute should have it listed as an attribute, i.e., when searching for a class with that attribute we only want to retrieve the top-most class(es) in the hierarchy which define it, and not every (sub)class that inherited it.

Return type:

None

subtype_distance(supertype, subtype)

Computes the number of subclassing steps from the supertype to the subtype.

The number of subclassing steps is the shortest path length in the inheritance graph from the supertype to the subtype. The length from a type to itself is 0. If there is no path from the supertype to the subtype, i.e. if the supertype is not a supertype of the subtype, the method returns None. Similar to is_maybe_subtype, for unions it is sufficient that one type of the Union is a subtype and further the shortest path between all arguments of the Union is computed. See _SubtypeDistanceVisitor for special cases.

Examples

Assume (Super > Sub > SubSub). Super, Sub -> 1 Sub, Super -> None Super, Super -> 0 Super, SubSub -> 2

Parameters:

• supertype (ProperType) – The more general type.

• subtype (ProperType) – The more specific type.

Return type:

int | None

Returns:

The number of subclassing steps from left to right, or None if no path exists.

to_type_info(typ)

Find or create TypeInfo for the given type.

Parameters:

typ – The raw type we want to convert.

Return type:

TypeInfo

Returns:

A TypeInfo object.

try_to_load_type(candidate, globs)

Try to load the given type.

Parameters:

• candidate (str) – The type to load

• globs – The globals that should be used for loading.

Return type:

ProperType

Returns:

The loaded type or Any.

static type_hints_provider(method)

Provides PEP484-style type information, if available.

Parameters:

method (Callable) – The method for which we want type hints.

Return type:

dict[str, Any]

Returns:

A dict mapping parameter names to type hints.

wrap_var_param_type(typ, param_kind)

Wrap parameter types.

Wrap the parameter type of *args and **kwargs in List[…] or Dict[str, …], respectively.

Parameters:

• typ (ProperType) – The type to be wrapped.

• param_kind – the kind of parameter.

Return type:

ProperType

Returns:

The wrapped type, or the original type, if no wrapping is required.

class pynguin.analyses.typesystem.TypeVisitor

A type visitor.

Note that the parameter of the visit_* methods is called ‘left’, because it makes the implementations of _SubTypeVisitor and _MaybeSubTypeVisitor more clear and Python does not like changing the names of parameters in subclasses, thus we renamed them in this class.

abstract visit_any_type(left)

Visit the Any type.

Parameters:

left (AnyType) – the Any type

Return type:

TypeVar(T)

Returns:

result of the visit

abstract visit_instance(left)

Visit an instance.

Parameters:

left (Instance) – instance

Return type:

TypeVar(T)

Returns:

result of the visit

abstract visit_none_type(left)

Visit the None type.

Parameters:

left (NoneType) – the None type

Return type:

TypeVar(T)

Returns:

result of the visit

visit_string_subtype(left)

Visit a string subtype.

Parameters:

left (StringSubtype) – string subtype

Return type:

TypeVar(T)

Returns:

Result of the visit

abstract visit_tuple_type(left)

Visit a tuple type.

Parameters:

left (TupleType) – tuple

Return type:

TypeVar(T)

Returns:

result of the visit

abstract visit_union_type(left)

Visit a union.

Parameters:

left (UnionType) – union

Return type:

TypeVar(T)

Returns:

result of the visit

abstract visit_unsupported_type(left)

Visit unsupported type.

Parameters:

left (Unsupported) – unsupported

Return type:

TypeVar(T)

Returns:

result of the visit

class pynguin.analyses.typesystem.URLString

A string that represents a URL.

Examples: “http://example.com”, “https://www.domain.org/path?query=param”.

class pynguin.analyses.typesystem.UUIDString

A string that represents a UUID.

Examples: “123e4567-e89b-12d3-a456-426614174000”.

class pynguin.analyses.typesystem.UnionType(items)

The union type Union[T1, …, Tn] (at least one type argument).

accept(visitor)

Accept a type visitor.

Parameters:

visitor (TypeVisitor[TypeVar(T)]) – the visitor

Return type:

TypeVar(T)

class pynguin.analyses.typesystem.Unsupported

Marks an unsupported type in the type system.

Artificial type which represents a type that is currently not supported by our type abstraction. This is purely used for statistic purposes and should not be encountered during regular use.

accept(visitor)

Accept a type visitor.

Parameters:

visitor (TypeVisitor[TypeVar(T)]) – the visitor

Return type:

TypeVar(T)

pynguin.testcase.testcase

Provides a libcst-backed test case implementation.

class pynguin.testcase.testcase.MLStatementInfo(kind, dtype=None, low=None, high=None, is_tuple=False, allowed_values=None)

Metadata describing an ML-specific statement.

There are no statement subclasses, so ML-specific statements (ndarray literals, dtype picks, np.array/tensor-constructor calls) are plain Statement instances carrying this metadata instead. The kind field distinguishes the ndarray, allowed-values, ML-scalar, and ML-call variants.

allowed_values: list[int | float | bool | str] | None = None

The pool of values an allowed_values statement was picked from.

dtype: str | None = None

Numpy dtype name, e.g. "int32", for ndarray/ml_scalar kinds.

high: float | None = None

Upper bound of the value range used to generate the payload.

is_tuple: bool = False

Whether a 1-D ndarray payload is rendered/mutated as a tuple.

low: float | None = None

Lower bound of the value range used to generate the payload.

class pynguin.testcase.testcase.Statement(node, bound_variable=None, bound_type=None, assertions=<factory>, accessible=None, ml_info=None, local_search_applied=False)

Wraps a single libcst statement node.

has_only_exception_assertion()

Does this statement only have an exception assertion?

Return type:

bool

Returns:

True, if there is only an exception assertion.

local_search_applied: bool = False

Whether local search already tried the same-datatype strategy on this exact value once, used to decide when to escape a local optimum by randomizing the value first. Not carried over by TestCase.clone() or TestCase.append_test_case_from() (both rebuild Statement instances without passing it), so it resets to False on clone – a deliberate, slightly more exploratory choice.

used_variables()

Return (and cache) the set of variable names read by this statement.

The result is computed lazily on first call and cached, since CST nodes are immutable and Statement.node is never mutated after construction.

Return type:

frozenset[str]

Returns:

The frozenset of variable names read by this statement.

class pynguin.testcase.testcase.TestCase

An ordered list of CST-backed statements forming a single test case.

add_statement(stmt)

Append a statement and update the type registry.

Parameters:

stmt (Statement) – The statement to append.

Return type:

None

append_test_case(other)

Append a copy of other’s statements, renaming variables to fresh names.

Variables bound by other are renamed to freshly allocated var_N names in this test case to avoid collisions with existing names.

Parameters:

other (TestCase) – The test case whose statements should be appended.

Return type:

None

append_test_case_from(other, start)

Append other’s statements from start onwards, renaming variables.

Like append_test_case() but begins at start instead of 0. Variables from other[start:] are still renamed consistently so that references between those statements are preserved and do not collide with names already present in this test case.

Tail statements may also read variables that other bound before start. Such free references are remapped to a type-compatible variable already present in this test case; when none exists, the statement (and every tail statement transitively depending on it) is dropped instead of being appended with a dangling name.

This is the correct building block for single-point crossover: the second parent’s tail is appended with fresh variable names, avoiding silent overwrites of names introduced by the first parent and NameErrors from references into the second parent’s head.

Parameters:

• other (TestCase) – The test case to read statements from.

• start (int) – First index of other to include (inclusive).

Return type:

None

chop(position)

Remove all statements after position (keeping 0..position).

Parameters:

position (int) – The index of the last statement to keep.

Return type:

None

clone()

Return a deep copy; mutations to the clone do not affect the original.

Return type:

TestCase

Returns:

A deep copy of this test case.

forward_dependencies(index)

Return index plus every later statement transitively depending on it.

Dependencies are computed name-based: a later statement depends on the statement at index if it (transitively) reads a variable bound by a statement already in the dependency set.

Parameters:

index (int) – The index of the root statement.

Return type:

set[int]

Returns:

The set of statement indices in the forward-dependency closure (including index itself).

get_assertions()

Return all assertions across all statements.

Return type:

list[Assertion]

Returns:

The list of all assertions in this test case.

get_statement(index)

Return the statement at index.

Parameters:

index (int) – The index of the statement.

Return type:

Statement

Returns:

The statement at the index.

insert_statement(index, stmt)

Insert a statement at index and rebuild the type registry.

Parameters:

• index (int) – The index to insert the statement at.

• stmt (Statement) – The statement to insert.

Return type:

None

next_var_name()

Allocate and return the next var_N name.

Return type:

str

Returns:

The next variable name.

remove_statement(index)

Remove and return the statement at index; rebuild the type registry.

Parameters:

index (int) – The index of the statement to remove.

Return type:

Statement

Returns:

The removed statement.

remove_statement_with_forward_dependencies(index)

Remove the statement at index and all of its forward dependencies.

Parameters:

index (int) – The index of the statement to remove.

Return type:

set[int]

Returns:

The set of removed statement indices.

remove_statements_batch(indices)

Remove all statements at indices in one pass; rebuild registry once.

Parameters:

indices (set[int]) – The set of indices to remove.

Return type:

None

remove_unused_variables()

Remove assignments to variables that are not used later in the test case.

Uses a backward pass to track alive variables. For each assignment, if the bound variable is not alive at that point, the assignment is replaced with a simple expression statement.

Return type:

None

replace_statement(index, stmt)

Replace the statement at index in-place; rebuild the type registry.

Parameters:

• index (int) – The index of the statement to replace.

• stmt (Statement) – The replacement statement.

Return type:

None

size()

Return the number of statements.

Return type:

int

Returns:

The number of statements.

size_with_assertions()

Return the number of statements plus the total number of assertions.

Return type:

int

Returns:

The number of statements plus assertions.

statements()

Return a shallow copy of the statement list.

Return type:

list[Statement]

Returns:

A shallow copy of the statement list.

to_code()

Return the Python source string for this test case.

Return type:

str

Returns:

The source string for this test case.

to_module()

Return all statements as a flat cst.Module.

Return type:

Module

Returns:

The module containing all statements.

to_test_function(index=0)

Wrap all statements in def test_N(): and return as a Module.

Parameters:

index (int) – The index used to name the test function.

Return type:

Module

Returns:

The module containing the test function.

variables_of_type(t)

Return all variable names currently bound to type t.

Parameters:

t (type) – The type to look up.

Return type:

list[str]

Returns:

The list of variable names bound to the type.

pynguin.ga.testcasefactory

Provides a factories for generating different kind of test cases.

class pynguin.ga.testcasefactory.RandomLengthTestCaseFactory(test_factory, test_cluster)

Create random test cases with random length.

get_test_case()

Retrieve a test case.

Returns:

DAR202

Return type:

A test case # noqa

class pynguin.ga.testcasefactory.SeededTestCaseFactory(delegate, population_provider)

Factory for getting seeded test cases.

With a certain probability a seeded testcase is returned instead of a randomly generated one. If a seeded testcase is returned, it is taken randomly from the pool of seeded testcases. If a randomly generated testcase is returned, the generation is delegated to the RandomLengthTestCaseFactory.

get_test_case()

Retrieve a test case.

Returns:

DAR202

Return type:

A test case # noqa

class pynguin.ga.testcasefactory.TestCaseFactory(test_factory)

Abstract class for test case factories.

abstract get_test_case()

Retrieve a test case.

Returns:

DAR202

Return type:

A test case # noqa

pynguin.testcase.execution

Contains all code related to test-case execution.

class pynguin.testcase.execution.AbstractTestCaseExecutor

Interface for a test case executor.

abstract add_observer(observer)

Add an execution observer.

Parameters:

observer (ExecutionObserver) – the observer to be added.

Return type:

None

abstract add_remote_observer(remote_observer)

Add a remote execution observer.

Parameters:

remote_observer (RemoteExecutionObserver) – the remote observer to be added.

Return type:

None

abstract clear_observers()

Remove all existing observers.

Return type:

None

abstract clear_remote_observers()

Remove all existing remote observers.

Return type:

None

abstract execute(test_case)

Executes all statements of the given test case.

Parameters:

test_case (TestCase) – the test case that should be executed.

Raises:

RuntimeError – If something goes wrong inside Pynguin during execution.

Return type:

ExecutionResult

Returns:

Result of the execution

execute_multiple(test_cases)

Executes multiple test cases.

Parameters:

test_cases (Iterable[TestCase]) – The test cases that should be executed.

Raises:

RuntimeError – If something goes wrong inside Pynguin during execution.

Yields:

The results of the execution

Return type:

Iterable[ExecutionResult]

abstract property module_provider: ModuleProvider

The module provider used by this executor.

Returns:

The used module provider

abstract property subject_properties: SubjectProperties

Provide access to the subject properties.

Returns:

The subject properties

abstract temporarily_add_observer(observer)

Temporarily add the given observer.

Parameters:

observer (ExecutionObserver) – The observer to add.

Return type:

AbstractContextManager[None]

abstract temporarily_add_remote_observer(remote_observer)

Temporarily add a remote observer.

Parameters:

remote_observer (RemoteExecutionObserver) – The remote observer to add.

Return type:

AbstractContextManager[None]

class pynguin.testcase.execution.ExecutionObserver

An observer that can be used to observe the execution of a test case.

abstract after_remote_test_case_execution(test_case, result)

Called after test case execution from the main thread.

Note: This method is always called, though the data you expect in the execution might not be there, if the execution of the test case timed out. You are not allowed to access thread local state here (due to how threading.local works, it isn’t even possible ;)), but you can do some postprocessing with the data from the execution result here.

Parameters:

• test_case (TestCase) – The test cases that was executed

• result (ExecutionResult) – The execution result

Return type:

None

abstract before_remote_test_case_execution(test_case)

Called before test case execution from the main thread.

Note: This method can be called with several test cases before the after_remote_test_case_execution method is called.

Parameters:

test_case (TestCase) – The test cases that will be executed.

Return type:

None

abstract property remote_observer: RemoteExecutionObserver

The remote observer.

Returns:

The remote observer

class pynguin.testcase.execution.ExecutionResult(timeout=False)

Result of an execution.

delete_statement_data(deleted_statements)

Delete statements at given indices.

It may happen that the test case is modified after execution, for example, by removing unused primitives. We have to update the execution result to reflect this, otherwise the indexes maybe wrong.

Parameters:

deleted_statements (set[int]) – The indexes of the deleted statements

Return type:

None

get_first_position_of_thrown_exception()

Provide the index of the first thrown exception or None.

Return type:

int | None

Returns:

The index of the first thrown exception, if any

has_test_exceptions()

Returns true if any exceptions were thrown during the execution.

Return type:

bool

Returns:

Whether the test has exceptions

report_new_thrown_exception(stmt_idx, ex)

Report an exception that was thrown during execution.

Parameters:

• stmt_idx (int) – the index of the statement, that caused the exception

• ex (BaseException) – the exception

Return type:

None

static shift_dict(to_shift, deleted_indexes)

Shifts the entries in the given dictionary.

Compute the entries’ new positions after the given statements were deleted.

Parameters:

• to_shift (dict[int, TypeVar(T)]) – The dict to shift

• deleted_indexes (set[int]) – A set of deleted statement indexes.

Return type:

dict[int, TypeVar(T)]

Returns:

The shifted dict

class pynguin.testcase.execution.ModuleProvider

Class for providing modules.

add_mutated_version(module_name, mutated_module)

Adds a mutated version of a module to the collection of mutated modules.

Parameters:

• module_name (str) – for the module name of the module, which should be mutated.

• mutated_module (ModuleType) – the custom module, which should be used.

Return type:

None

clear_mutated_modules()

Clear the existing aliases.

get_module(module_name)

Provides a module.

Either from sys.modules or if a mutated version for the given module name exists then the mutated version of the module will be returned.

Parameters:

module_name (str) – string for the module alias, which should be loaded

Raises:

ModuleNotImportedError – If the module is not imported.

Return type:

ModuleType

Returns:

the module which should be loaded.

class pynguin.testcase.execution.OutputSuppressionContext

A context manager that suppresses stdout and stderr.

Operates at two levels:

• Python level: redirects sys.stdout / sys.stderr to /dev/null.

• OS level: saves file descriptors 0/1/2 via os.dup so that if the SUT closes them (e.g. with open(1, 'w') where the int happens to be a stdio fd), they are restored on exit.

restore()

Restore stdout and stderr at both Python and OS level.

Return type:

None

class pynguin.testcase.execution.PatchRandomOnUnpickle

A hook that patches random when unpickled in a subprocess.

This ensures that random.Random.seed is patched before the SUT is unpickled and potentially creates new random.Random instances (e.g., mimesis).

class pynguin.testcase.execution.RemoteAssertionExecutionObserver

A remote observer which executes the assertions of statements.

Executes each statement’s assertions right after the statement, with checked-coverage instrumentation enabled (when the executor was configured for it via set_instrument(True)), and records their trace positions via the instrumentation tracer so they can later be sliced.

Enables slicing on the recorded data.

Note: this observer must only be registered together with executor.set_instrument(True); otherwise the assertion code executes uninstrumented and ExecutionTracer.track_assertion_position will fail to find a boolean-jump instruction to record.

after_statement_execution(statement, executor, namespace, exception)

Execute the statement’s assertions, instrumented, and track them.

Parameters:

• statement (Statement) – The statement that was executed.

• executor (TestCaseExecutor) – The executor that executed the statement.

• namespace (dict[str, Any]) – The shared namespace the statement executed in.

• exception (BaseException | None) – The exception raised by the statement, if any.

Return type:

None

after_test_case_execution(executor, test_case, result)

Not used.

Parameters:

• executor (TestCaseExecutor) – Not used

• test_case (TestCase) – Not used

• result (ExecutionResult) – Not used

Return type:

None

before_test_case_execution(test_case)

Not used.

Parameters:

test_case (TestCase) – not used

class pynguin.testcase.execution.RemoteExecutionObserver

A remote observer that can be used to observe the execution of a test case.

Important Note: If an observer is stateful, then this state must be encapsulated in a threading.local, i.e., be bound to a thread. Note that thread local data is initialized per thread, so there is no need to clear any pre-existing data (because there is none), as every thread gets its own instance.

Methods in this class are not allowed to interact with the ‘outside’ because this class could be sent to a remote environment. The only thing that should leave an observer are results when they are written to the execution result in RemoteExecutionObserver::after_test_case_execution.

You may interact with the ‘outside’ in ExecutionObserver::after_remote_test_case_execution.

Note: Usage of threading.local may interfere with debugging tools, such as pydevd. In such a case, disable Cython by setting the following environment variable: PYDEVD_USE_CYTHON=NO

For more details, look at some implementations, e.g., AssertionTraceObserver.

after_statement_execution(statement, executor, namespace, exception)

Called after a single statement is executed.

No-op by default; observers that need per-statement observation should override this.

Parameters:

• statement (Statement) – The statement that was executed.

• executor (TestCaseExecutor) – The executor that executed the statement.

• namespace (dict[str, Any]) – The shared namespace the statement executed in.

• exception (BaseException | None) – The exception raised by the statement, if any.

Return type:

None

abstract after_test_case_execution(executor, test_case, result)

Called after test case execution.

The call happens from the remote environment that executed the test case. You should override this method to extract information from the thread local storage to the execution result.

Note: When a timeout occurs, then this method might not be called at all.

Parameters:

• executor (TestCaseExecutor) – The executor that executed the test case

• test_case (TestCase) – The test cases that was executed

• result (ExecutionResult) – The execution result

Return type:

None

before_statement_execution(statement, node, namespace)

Called before a single statement is executed.

Returns node unchanged by default; observers that need to rewrite the node before it is executed (e.g. to inject proxies) should override this and return the modified node. The (possibly modified) node returned here is what actually gets executed, so overriders must return a valid node.

Parameters:

• statement (Statement) – The statement that is about to be executed.

• node (SimpleStatementLine | BaseCompoundStatement) – The CST node that is about to be executed for this statement.

• namespace (dict[str, Any]) – The shared namespace the statement will execute in.

Return type:

SimpleStatementLine | BaseCompoundStatement

Returns:

The CST node to execute (node unchanged by default).

abstract before_test_case_execution(test_case)

Called before test case execution.

Parameters:

test_case (TestCase) – The test cases that will be executed.

property state: dict[str, Any]

The state of the observer.

Returns:

The state of the observer

class pynguin.testcase.execution.RemoteReturnTypeObserver

An observer which observes the runtime types seen during execution.

class RemoteReturnTypeLocalState

Encapsulate observed return types.

after_statement_execution(statement, executor, namespace, exception)

Called after a single statement is executed.

No-op by default; observers that need per-statement observation should override this.

Parameters:

• statement (Statement) – The statement that was executed.

• executor (TestCaseExecutor) – The executor that executed the statement.

• namespace (dict[str, Any]) – The shared namespace the statement executed in.

• exception (BaseException | None) – The exception raised by the statement, if any.

Return type:

None

after_test_case_execution(executor, test_case, result)

Called after test case execution.

The call happens from the remote environment that executed the test case. You should override this method to extract information from the thread local storage to the execution result.

Note: When a timeout occurs, then this method might not be called at all.

Parameters:

• executor (TestCaseExecutor) – The executor that executed the test case

• test_case (TestCase) – The test cases that was executed

• result (ExecutionResult) – The execution result

before_test_case_execution(test_case)

Not used.

Parameters:

test_case (TestCase) – Not used

property state: dict[str, Any]

The state of the observer.

Returns:

The state of the observer

class pynguin.testcase.execution.RemoteTypeTracingObserver

A remote observer used for type tracing.

class RemoteTypeTracingLocalState

Thread local data for type tracing.

after_statement_execution(statement, executor, namespace, exception)

Called after a single statement is executed.

No-op by default; observers that need per-statement observation should override this.

Parameters:

• statement (Statement) – The statement that was executed.

• executor (TestCaseExecutor) – The executor that executed the statement.

• namespace (dict[str, Any]) – The shared namespace the statement executed in.

• exception (BaseException | None) – The exception raised by the statement, if any.

Return type:

None

after_test_case_execution(executor, test_case, result)

Called after test case execution.

The call happens from the remote environment that executed the test case. You should override this method to extract information from the thread local storage to the execution result.

Note: When a timeout occurs, then this method might not be called at all.

Parameters:

• executor (TestCaseExecutor) – The executor that executed the test case

• test_case (TestCase) – The test cases that was executed

• result (ExecutionResult) – The execution result

Return type:

None

before_statement_execution(statement, node, namespace)

Called before a single statement is executed.

Returns node unchanged by default; observers that need to rewrite the node before it is executed (e.g. to inject proxies) should override this and return the modified node. The (possibly modified) node returned here is what actually gets executed, so overriders must return a valid node.

Parameters:

• statement (Statement) – The statement that is about to be executed.

• node (SimpleStatementLine | BaseCompoundStatement) – The CST node that is about to be executed for this statement.

• namespace (dict[str, Any]) – The shared namespace the statement will execute in.

Return type:

SimpleStatementLine | BaseCompoundStatement

Returns:

The CST node to execute (node unchanged by default).

before_test_case_execution(test_case)

Not used.

Parameters:

test_case (TestCase) – Not used

property state: dict[str, Any]

The state of the observer.

Returns:

The state of the observer

class pynguin.testcase.execution.ReturnTypeObserver(test_cluster)

Observes the runtime types seen during execution.

Updates the return types of the called function with the observed types.

after_remote_test_case_execution(test_case, result)

Called after test case execution from the main thread.

Note: This method is always called, though the data you expect in the execution might not be there, if the execution of the test case timed out. You are not allowed to access thread local state here (due to how threading.local works, it isn’t even possible ;)), but you can do some postprocessing with the data from the execution result here.

Parameters:

• test_case (TestCase) – The test cases that was executed

• result (ExecutionResult) – The execution result

before_remote_test_case_execution(test_case)

Not used.

Parameters:

test_case (TestCase) – Not used

Return type:

None

property remote_observer: RemoteExecutionObserver

The remote observer.

Returns:

The remote observer

class pynguin.testcase.execution.SubprocessTestCaseExecutor(subject_properties, module_provider=None, maximum_test_execution_timeout=5, test_execution_time_per_statement=1)

An executor that executes the generated test cases in a subprocess.

class ConnectionStatus(value)

Status of the connection to the subprocess.

class SubprocessResultContext(test_cases_tuple, references_bindings, process, receiving_connection, connection_status, remote_observers)

Context for processing subprocess results.

execute(test_case)

Executes all statements of the given test case.

Parameters:

test_case (TestCase) – the test case that should be executed.

Raises:

RuntimeError – If something goes wrong inside Pynguin during execution.

Return type:

ExecutionResult

Returns:

Result of the execution

execute_multiple(test_cases)

Executes multiple test cases.

Parameters:

test_cases (Iterable[TestCase]) – The test cases that should be executed.

Raises:

RuntimeError – If something goes wrong inside Pynguin during execution.

Yields:

The results of the execution

Return type:

Iterable[ExecutionResult]

execute_with_exit_code(test_case)

Execute a test case in a subprocess and return the exit code.

This method executes a single test case in a separate subprocess and returns the exit code of the subprocess. If the subprocess crashes or times out, it returns None or a non-zero exit code.

Parameters:

test_case (TestCase) – The test case to execute

Return type:

int | None

Returns:

The exit code of the subprocess. A None or non-zero exit code indicates a crash.

class pynguin.testcase.execution.TestCaseExecutor(subject_properties, module_provider=None, maximum_test_execution_timeout=5, test_execution_time_per_statement=1)

An executor that executes the generated test cases.

add_observer(observer)

Add an execution observer.

Parameters:

observer (ExecutionObserver) – the observer to be added.

Return type:

None

add_remote_observer(remote_observer)

Add a remote execution observer.

Parameters:

remote_observer (RemoteExecutionObserver) – the remote observer to be added.

Return type:

None

clear_observers()

Remove all existing observers.

Return type:

None

clear_remote_observers()

Remove all existing remote observers.

Return type:

None

execute(test_case)

Executes all statements of the given test case.

Parameters:

test_case (TestCase) – the test case that should be executed.

Raises:

RuntimeError – If something goes wrong inside Pynguin during execution.

Return type:

ExecutionResult

Returns:

Result of the execution

execute_source(code_str, namespace)

Compile and execute source code against the shared namespace.

Compiles with AST_FILENAME (matching the filename that the whole instrumentation/slicing stack special-cases for test code, see pynguin.instrumentation.AST_FILENAME) and, if this executor was configured for checked-coverage instrumentation (set_instrument(True)), applies self._checked_transformer before executing, so that every executed instruction is appended to the trace and can later be used as a dynamic-slicing criterion.

Parameters:

• code_str (str) – The source code to compile and execute.

• namespace (dict[str, Any]) – The shared namespace (used as both globals and locals).

Return type:

BaseException | None

Returns:

The raised exception, if any, otherwise None.

property module_provider: ModuleProvider

The module provider used by this executor.

Returns:

The used module provider

register_crash_revealing_hash(test_case_hash)

Record a crash-revealing test-case hash.

Parameters:

test_case_hash (str) – The hash of the crashing test case.

Return type:

int | None

Returns:

The new number of distinct crash-revealing hashes if the hash was not seen before, otherwise None.

set_instrument(instrument)

Set if the test is to be instrumented as well.

Parameters:

instrument (bool) – Whether to instrument the test and its assertions.

Return type:

None

property subject_properties: SubjectProperties

Provide access to the subject properties.

Returns:

The subject properties

temporarily_add_observer(observer)

Temporarily add the given observer.

Parameters:

observer (ExecutionObserver) – The observer to add.

Return type:

Generator[None, None, None]

temporarily_add_remote_observer(remote_observer)

Temporarily add a remote observer.

Parameters:

remote_observer (RemoteExecutionObserver) – The remote observer to add.

Return type:

Generator[None, None, None]

class pynguin.testcase.execution.TypeTracingObserver(cluster)

An execution observer used for type tracing.

It wraps parameters in proxies in order to make better guesses on their type.

after_remote_test_case_execution(test_case, result)

Called after test case execution from the main thread.

Note: This method is always called, though the data you expect in the execution might not be there, if the execution of the test case timed out. You are not allowed to access thread local state here (due to how threading.local works, it isn’t even possible ;)), but you can do some postprocessing with the data from the execution result here.

Parameters:

• test_case (TestCase) – The test cases that was executed

• result (ExecutionResult) – The execution result

Return type:

None

before_remote_test_case_execution(test_case)

Not used.

Parameters:

test_case (TestCase) – Not used

Return type:

None

property remote_observer: RemoteTypeTracingObserver

The remote observer.

Returns:

The remote observer

class pynguin.testcase.execution.TypeTracingTestCaseExecutor(delegate, cluster, type_tracing_probability=1.0)

A test case executor that delegates to another executor.

Every test case is executed twice, one time for the regular result and one time with proxies in order to refine parameter types.

add_observer(observer)

Add an execution observer.

Parameters:

observer (ExecutionObserver) – the observer to be added.

Return type:

None

add_remote_observer(remote_observer)

Add a remote execution observer.

Parameters:

remote_observer (RemoteExecutionObserver) – the remote observer to be added.

Return type:

None

clear_observers()

Remove all existing observers.

Return type:

None

clear_remote_observers()

Remove all existing remote observers.

Return type:

None

execute(test_case)

Executes all statements of the given test case.

Parameters:

test_case (TestCase) – the test case that should be executed.

Raises:

RuntimeError – If something goes wrong inside Pynguin during execution.

Return type:

ExecutionResult

Returns:

Result of the execution

property module_provider: ModuleProvider

The module provider used by this executor.

Returns:

The used module provider

property subject_properties: SubjectProperties

Provide access to the subject properties.

Returns:

The subject properties

temporarily_add_observer(observer)

Temporarily add the given observer.

Parameters:

observer (ExecutionObserver) – The observer to add.

Return type:

AbstractContextManager[None]

temporarily_add_remote_observer(remote_observer)

Temporarily add a remote observer.

Parameters:

remote_observer (RemoteExecutionObserver) – The remote observer to add.

Return type:

AbstractContextManager[None]

pynguin.testcase.execution.find_call(node)

Return the top-level call of a statement node, if it wraps one.

Covers var_0 = module_.f(...), var_0 = var_1.m(...) and demoted module_.f(...) expression statements.

Parameters:

node (SimpleStatementLine | BaseCompoundStatement) – The statement’s CST node.

Return type:

Call | None

Returns:

The wrapped cst.Call, or None if the node does not wrap one.

pynguin.testcase.execution.map_args_to_params(call, signature)

Map each call argument to the parameter it binds.

Keyword args resolve by name; positional args consume the signature’s positional parameters (POSITIONAL_ONLY / POSITIONAL_OR_KEYWORD, skipping a leading self) in declaration order. *args / **kwargs expansions and unresolvable keyword args are skipped.

Parameters:

• call (Call) – The call whose arguments to map.

• signature (Signature) – The callable’s inferred signature.

Return type:

list[tuple[int, str, Parameter]]

Returns:

A list of (arg_index, param_name, parameter) tuples.

pynguin.testcase.execution.suppress_logging()

Suppress all log messages during SUT execution.

Yields:

Nothing; restores logging on exit.

pynguin.instrumentation.tracer

Contains all code related to test-case execution.

class pynguin.instrumentation.tracer.AbstractExecutionTracer

An abstract execution tracer.

The results are stored in an execution trace.

abstract check()

Check if the thread that called this method should still be running.

Raises:

RuntimeError – if the thread is not running anymore.

Return type:

None

abstract disable()

Disable tracing.

Return type:

None

abstract enable()

Enable tracing.

Return type:

None

abstract executed_bool_predicate(value, predicate)

A predicate that is based on a boolean value was executed.

Parameters:

• value – the value

• predicate (int) – the predicate identifier

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract executed_code_object(code_object_id)

Mark a code object as executed.

This means, that the routine which refers to this code object was at least called once.

Parameters:

code_object_id (int) – the code object id to mark

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract executed_compare_predicate(value1, value2, predicate, cmp_op)

A predicate that is based on a comparison was executed.

Parameters:

• value1 – the first value

• value2 – the second value

• predicate (int) – the predicate identifier

• cmp_op (PynguinCompare) – the compare operation

Raises:

• RuntimeError – raised when called from another thread.

• AssertionError – when encountering an unknown compare op.

Return type:

None

abstract executed_exception_match(err, exc, predicate)

A predicate that is based on exception matching was executed.

Parameters:

• err (BaseException | type[BaseException]) – The raised exception

• exc (type[BaseException]) – The matching condition

• predicate (int) – the predicate identifier

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract executed_in_presence_predicate(value1, value2, predicate)

An auxiliary membership predicate was executed.

Computes a guided branch distance for value1 in value2 with an execution-time guard on the container size to limit overhead.

Parameters:

• value1 – The prospective key/index to check membership for.

• value2 – The container to check membership in.

• predicate (int) – The predicate identifier.

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract get_trace()

Get the trace with the current information.

Return type:

ExecutionTrace

Returns:

The current execution trace

abstract property import_trace: ExecutionTrace

The trace that was generated when the SUT was imported.

Returns:

The execution trace after executing the import statements

abstract init_trace()

Create a new trace that only contains the trace data from the import.

Return type:

None

abstract is_disabled()

Should we track anything?

We might have to disable tracing, e.g. when calling __eq__ ourselves. Otherwise, we create an endless recursion.

Return type:

bool

Returns:

Whether we should track anything

abstract reset()

Resets everything.

Should be called before instrumentation. Clears all data, so we can handle a reload of the SUT.

Return type:

None

abstract stop()

Stop the tracer.

This should be called when the tracer is no longer needed, e.g., when the test case execution is finished.

Return type:

None

abstract store_import_trace()

Stores the current trace as the import trace.

Should only be done once, after a module was loaded. The import trace will be merged into every subsequently recorded trace.

Return type:

None

temporarily_disable()

Temporarily disable tracing.

If the tracing is already disabled, do nothing.

Return type:

Generator[None, None, None]

temporarily_enable()

Temporarily enable tracing.

If the tracing is already enabled, do nothing.

Return type:

Generator[None, None, None]

abstract track_assertion_position(assertion)

Track the position of an assertion in the trace.

Parameters:

assertion (Assertion) – the assertion of the statement

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract track_attribute_access(module, code_object_id, node_id, opcode, lineno, offset, attr_name, obj)

Track an attribute access instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• attr_name (str | None) – the name of the accessed attribute

• obj (object) – the object containing the accessed attribute

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract track_call(module, code_object_id, node_id, opcode, lineno, offset, arg)

Track a method call instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• arg (int) – the argument used in the method call

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract track_exception_assertion(statement)

Track the position of an exception assertion in the trace.

Normally, to track an assertion, we trace the POP_JUMP_IF_TRUE instruction contained by each assertion. The pytest exception assertion does not use an assertion containing this instruction. Therefore, we trace the instruction that was last executed before the exception.

Parameters:

statement (Statement) – the statement causing the exception

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract track_generic(module, code_object_id, node_id, opcode, lineno, offset)

Track a generic instruction inside the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract track_jump(module, code_object_id, node_id, opcode, lineno, offset, target_id)

Track a jump instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• target_id (int) – the offset of the target of the jump

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract track_line_visit(line_id)

Tracks the visit of a line.

Parameters:

line_id (int) – the if of the line that was visited

Raises:

RuntimeError – raised when called from another thread

Return type:

None

abstract track_memory_access(module, code_object_id, node_id, opcode, lineno, offset, var_name, var_value)

Track a memory access instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• var_name (str | CellVar | FreeVar | tuple[str | CellVar | FreeVar, str | CellVar | FreeVar]) – the used variable name

• var_value (object) – the value stored in the used variable

Raises:

• ValueError – when no argument is given

• RuntimeError – raised when called from another thread

Return type:

None

abstract track_return(module, code_object_id, node_id, opcode, lineno, offset)

Track a return instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

Raises:

RuntimeError – raised when called from another thread

Return type:

None

class pynguin.instrumentation.tracer.CodeObjectMetaData(code_object, parent_code_object_id, cfg, cdg)

Stores meta data of a code object.

class pynguin.instrumentation.tracer.ExecutedAssertion(trace_position, assertion)

Data class for assertions of a testcase traced during execution for slicing.

class pynguin.instrumentation.tracer.ExecutionTrace(executed_code_objects=<factory>, executed_predicates=<factory>, true_distances=<factory>, false_distances=<factory>, covered_line_ids=<factory>, executed_instructions=<factory>, object_addresses=<factory>, executed_assertions=<factory>, checked_lines=<factory>)

Stores trace information about the execution.

add_attribute_instruction(module, code_object_id, node_id, opcode, lineno, offset, attr_name, src_address, arg_address, is_mutable_type, is_method)

Creates a new ExecutedAttributeInstruction object and adds it to the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• attr_name (str) – the name of the accessed attribute

• src_address (int) – the memory address of the attribute

• arg_address (int) – the memory address of the argument

• is_mutable_type (bool) – if the attribute is mutable

• is_method (bool) – if the attribute is a method

Return type:

None

add_call_instruction(module, code_object_id, node_id, opcode, lineno, offset, arg)

Creates a new ExecutedCallInstruction object and adds it to the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• arg (int) – the argument to the instruction

Return type:

None

add_instruction(module, code_object_id, node_id, opcode, lineno, offset)

Creates a new ExecutedInstruction object and adds it to the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

Return type:

None

add_jump_instruction(module, code_object_id, node_id, opcode, lineno, offset, target_id)

Creates a new ExecutedControlInstruction object and adds it to the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• target_id (int) – the target offset to jump to

Return type:

None

add_memory_instruction(module, code_object_id, node_id, opcode, lineno, offset, arg_name, arg_address, is_mutable_type, object_creation)

Creates a new ExecutedMemoryInstruction object and adds it to the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• arg_name (str | tuple[str, str]) – the name of the argument

• arg_address (int | tuple[int, int]) – the memory address of the argument

• is_mutable_type (bool | tuple[bool, bool]) – if the argument is mutable

• object_creation (bool | tuple[bool, bool]) – if the instruction creates the object used

Return type:

None

add_return_instruction(module, code_object_id, node_id, opcode, lineno, offset)

Creates a new ExecutedReturnInstruction object and adds it to the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

Return type:

None

property covered_code_objects: OrderedSet[int]

Alias for the set of executed (i.e. covered) code objects.

The libcst-based test case representation refers to covered code objects under this name; it maps onto executed_code_objects.

Returns:

The set of code object ids that were executed.

merge(other)

Merge the values from the other execution trace.

Parameters:

other (ExecutionTrace) – Merges the other traces into this trace

Return type:

None

update_predicate_distances(distance_true, distance_false, predicate)

Update the distances and predicate execution count.

Parameters:

• distance_true (float) – the measured true distance

• distance_false (float) – the measured false distance

• predicate (int) – the predicate id

Return type:

None

class pynguin.instrumentation.tracer.ExecutionTracer

Tracks branch distances and covered statements during execution.

The results are stored in an execution trace.

class TracerLocalState

Encapsulate state that is thread specific.

static attribute_lookup(object_type, attribute)

Check the dictionary of classes making up the MRO (method resolution order).

It is inspired by the _PyType_Lookup C function in CPython.

Parameters:

• object_type – The type object to check

• attribute (str) – the attribute to check for in the class. It must be a data descriptor to be prioritized here.

Return type:

int

Returns:

The id of the object type or the class if it has the attribute, -1 otherwise

check()

Check if the thread that called this method should still be running.

Raises:

RuntimeError – if the thread is not running anymore.

Return type:

None

disable()

Disable tracing.

Return type:

None

enable()

Enable tracing.

Return type:

None

executed_bool_predicate(value, predicate)

A predicate that is based on a boolean value was executed.

Parameters:

• value – the value

• predicate (int) – the predicate identifier

Raises:

RuntimeError – raised when called from another thread

Return type:

None

executed_code_object(code_object_id)

Mark a code object as executed.

This means, that the routine which refers to this code object was at least called once.

Parameters:

code_object_id (int) – the code object id to mark

Raises:

RuntimeError – raised when called from another thread

Return type:

None

executed_compare_predicate(value1, value2, predicate, cmp_op)

A predicate that is based on a comparison was executed.

Parameters:

• value1 – the first value

• value2 – the second value

• predicate (int) – the predicate identifier

• cmp_op (PynguinCompare) – the compare operation

Raises:

• RuntimeError – raised when called from another thread.

• AssertionError – when encountering an unknown compare op.

Return type:

None

executed_exception_match(err, exc, predicate)

A predicate that is based on exception matching was executed.

Parameters:

• err (BaseException | type[BaseException]) – The raised exception

• exc (type[BaseException]) – The matching condition

• predicate (int) – the predicate identifier

Raises:

RuntimeError – raised when called from another thread

Return type:

None

executed_in_presence_predicate(value1, value2, predicate)

Conditionally compute an auxiliary ‘IN’ predicate distance.

This helper provides guidance for subscripts like container[key] by reporting a membership distance key in container before the subscript executes. To control overhead, it only computes a distance when the container is sized and its size does not exceed max_container_size.

Parameters:

• value1 – The prospective key/index to look up (e.g., key).

• value2 – The container to check membership in (e.g., container).

• predicate (int) – The predicate id to update.

Return type:

None

get_trace()

Get the trace with the current information.

Return type:

ExecutionTrace

Returns:

The current execution trace

property import_trace: ExecutionTrace

The trace that was generated when the SUT was imported.

Returns:

The execution trace after executing the import statements

init_trace()

Create a new trace that only contains the trace data from the import.

Return type:

None

is_disabled()

Should we track anything?

We might have to disable tracing, e.g. when calling __eq__ ourselves. Otherwise, we create an endless recursion.

Return type:

bool

Returns:

Whether we should track anything

reset()

Resets everything.

Should be called before instrumentation. Clears all data, so we can handle a reload of the SUT.

Return type:

None

property state: dict

Get the current state.

Returns:

The current state

stop()

Stop the tracer.

This should be called when the tracer is no longer needed, e.g., when the test case execution is finished.

Return type:

None

store_import_trace()

Stores the current trace as the import trace.

Should only be done once, after a module was loaded. The import trace will be merged into every subsequently recorded trace.

Return type:

None

track_assertion_position(assertion)

Track the position of an assertion in the trace.

Parameters:

assertion (Assertion) – the assertion of the statement

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_attribute_access(module, code_object_id, node_id, opcode, lineno, offset, attr_name, obj)

Track an attribute access instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• attr_name (str | None) – the name of the accessed attribute

• obj (object) – the object containing the accessed attribute

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_call(module, code_object_id, node_id, opcode, lineno, offset, arg)

Track a method call instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• arg (int) – the argument used in the method call

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_exception_assertion(statement)

Track the position of an exception assertion in the trace.

Normally, to track an assertion, we trace the POP_JUMP_IF_TRUE instruction contained by each assertion. The pytest exception assertion does not use an assertion containing this instruction. Therefore, we trace the instruction that was last executed before the exception.

Parameters:

statement (Statement) – the statement causing the exception

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_generic(module, code_object_id, node_id, opcode, lineno, offset)

Track a generic instruction inside the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_jump(module, code_object_id, node_id, opcode, lineno, offset, target_id)

Track a jump instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• target_id (int) – the offset of the target of the jump

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_line_visit(line_id)

Tracks the visit of a line.

Parameters:

line_id (int) – the if of the line that was visited

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_memory_access(module, code_object_id, node_id, opcode, lineno, offset, var_name, var_value)

Track a memory access instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• var_name (str | CellVar | FreeVar | tuple[str | CellVar | FreeVar, str | CellVar | FreeVar]) – the used variable name

• var_value (object) – the value stored in the used variable

Raises:

• ValueError – when no argument is given

• RuntimeError – raised when called from another thread

Return type:

None

track_return(module, code_object_id, node_id, opcode, lineno, offset)

Track a return instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

Raises:

RuntimeError – raised when called from another thread

Return type:

None

class pynguin.instrumentation.tracer.InstrumentationExecutionTracer(tracer)

An InstrumentationExecutionTracer is a sort of proxy for an ExecutionTracer.

This was done because when a module is instrumented, instructions are inserted into its bytecode and refer directly to a tracer. This means that without the use of a proxy, it would be impossible to modify the tracer, as there are direct references between the bytecode instructions and the tracer. By adding a proxy between the bytecode and the tracer, this ensures that the bytecode only has direct references to the proxy but no references to the tracer, so the tracer can be modified without any problems.

check()

Check if the thread that called this method should still be running.

Raises:

RuntimeError – if the thread is not running anymore.

Return type:

None

disable()

Disable tracing.

Return type:

None

enable()

Enable tracing.

Return type:

None

executed_bool_predicate(value, predicate)

A predicate that is based on a boolean value was executed.

Parameters:

• value – the value

• predicate (int) – the predicate identifier

Raises:

RuntimeError – raised when called from another thread

Return type:

None

executed_code_object(code_object_id)

Mark a code object as executed.

This means, that the routine which refers to this code object was at least called once.

Parameters:

code_object_id (int) – the code object id to mark

Raises:

RuntimeError – raised when called from another thread

Return type:

None

executed_compare_predicate(value1, value2, predicate, cmp_op)

A predicate that is based on a comparison was executed.

Parameters:

• value1 – the first value

• value2 – the second value

• predicate (int) – the predicate identifier

• cmp_op (PynguinCompare) – the compare operation

Raises:

• RuntimeError – raised when called from another thread.

• AssertionError – when encountering an unknown compare op.

Return type:

None

executed_exception_match(err, exc, predicate)

A predicate that is based on exception matching was executed.

Parameters:

• err (BaseException | type[BaseException]) – The raised exception

• exc (type[BaseException]) – The matching condition

• predicate (int) – the predicate identifier

Raises:

RuntimeError – raised when called from another thread

Return type:

None

executed_in_presence_predicate(value1, value2, predicate)

An auxiliary membership predicate was executed.

Computes a guided branch distance for value1 in value2 with an execution-time guard on the container size to limit overhead.

Parameters:

• value1 – The prospective key/index to check membership for.

• value2 – The container to check membership in.

• predicate (int) – The predicate identifier.

Raises:

RuntimeError – raised when called from another thread

Return type:

None

get_trace()

Get the trace with the current information.

Return type:

ExecutionTrace

Returns:

The current execution trace

property import_trace: ExecutionTrace

The trace that was generated when the SUT was imported.

Returns:

The execution trace after executing the import statements

init_trace()

Create a new trace that only contains the trace data from the import.

Return type:

None

is_disabled()

Should we track anything?

We might have to disable tracing, e.g. when calling __eq__ ourselves. Otherwise, we create an endless recursion.

Return type:

bool

Returns:

Whether we should track anything

reset()

Resets everything.

Should be called before instrumentation. Clears all data, so we can handle a reload of the SUT.

Return type:

None

stop()

Stop the tracer.

This should be called when the tracer is no longer needed, e.g., when the test case execution is finished.

Return type:

None

store_import_trace()

Stores the current trace as the import trace.

Should only be done once, after a module was loaded. The import trace will be merged into every subsequently recorded trace.

Return type:

None

track_assertion_position(assertion)

Track the position of an assertion in the trace.

Parameters:

assertion (Assertion) – the assertion of the statement

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_attribute_access(module, code_object_id, node_id, opcode, lineno, offset, attr_name, obj)

Track an attribute access instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• attr_name (str | None) – the name of the accessed attribute

• obj (object) – the object containing the accessed attribute

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_call(module, code_object_id, node_id, opcode, lineno, offset, arg)

Track a method call instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• arg (int) – the argument used in the method call

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_exception_assertion(statement)

Track the position of an exception assertion in the trace.

Normally, to track an assertion, we trace the POP_JUMP_IF_TRUE instruction contained by each assertion. The pytest exception assertion does not use an assertion containing this instruction. Therefore, we trace the instruction that was last executed before the exception.

Parameters:

statement (Statement) – the statement causing the exception

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_generic(module, code_object_id, node_id, opcode, lineno, offset)

Track a generic instruction inside the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_jump(module, code_object_id, node_id, opcode, lineno, offset, target_id)

Track a jump instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• target_id (int) – the offset of the target of the jump

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_line_visit(line_id)

Tracks the visit of a line.

Parameters:

line_id (int) – the if of the line that was visited

Raises:

RuntimeError – raised when called from another thread

Return type:

None

track_memory_access(module, code_object_id, node_id, opcode, lineno, offset, var_name, var_value)

Track a memory access instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

• var_name (str | CellVar | FreeVar | tuple[str | CellVar | FreeVar, str | CellVar | FreeVar]) – the used variable name

• var_value (object) – the value stored in the used variable

Raises:

• ValueError – when no argument is given

• RuntimeError – raised when called from another thread

Return type:

None

track_return(module, code_object_id, node_id, opcode, lineno, offset)

Track a return instruction in the trace.

Parameters:

• module (str) – File name of the module containing the instruction

• code_object_id (int) – code object containing the instruction

• node_id (int) – the node of the code object containing the instruction

• opcode (int) – the opcode of the instruction

• lineno (int) – the line number of the instruction

• offset (int) – the offset of the instruction

Raises:

RuntimeError – raised when called from another thread

Return type:

None

class pynguin.instrumentation.tracer.LineMetaData(code_object_id, file_name, line_number)

Stores meta data of a line.

class pynguin.instrumentation.tracer.PredicateMetaData(line_no, code_object_id, node)

Stores meta data of a predicate.

class pynguin.instrumentation.tracer.SubjectProperties(instrumentation_tracer=<factory>, code_object_counter=<factory>, existing_code_objects=<factory>, existing_predicates=<factory>, existing_lines=<factory>)

Contains properties about the subject under test.

The subject properties are code objects, predicates and lines:

• Code Objects: Compiled chunks of code (functions, methods, modules). Tracked in CodeObjectMetaData with references to the compiled code, parent, control graphs, and a unique ID. Represent the program’s structural units.

• Predicates: Decision points within code objects (e.g., if, while). Tracked in PredicateMetaData with line number, owning code object, and graph node. Used for branch coverage and measuring branch distances.

• Lines: Individual lines of code within code objects. Tracked in LineMetaData with file name and line number. Used for measuring line coverage.

Example:

def example(x):
    if x > 0:  # Predicate
        return "pos"
    return "non-pos"

# The function ``example`` is a Code Object and ``x > 0`` is a Predicate.

property branch_less_code_objects: Iterable[int]

Get the existing code objects that do not contain a branch.

Every code object is initially seen as branch-less until a predicate is registered for it.

Returns:

The existing code objects that do not contain a branch.

create_code_object_id()

Create a new code object ID.

Return type:

int

Returns:

A new code object ID.

get_line_control_dependencies(line_id)

Provides the control dependencies for a given line.

Parameters:

line_id (int) – The line id

Return type:

tuple[bool, tuple[tuple[int, bool], ...]]

Returns:

A tuple of (is_root_dependent, tuple of (predicate_id, branch_value))

lineids_to_linenos(line_ids)

Convenience method to translate line ids to line numbers.

Parameters:

line_ids (OrderedSet[int]) – The ids that should be translated.

Return type:

OrderedSet[int]

Returns:

The line numbers.

register_code_object(code_object_id, meta)

Declare that a code object exists.

Parameters:

• code_object_id (int) – the id of the code object, which should be used to identify the object

• instrumentation. (during)

• meta (CodeObjectMetaData) – the code objects existing

Return type:

None

register_line(meta)

Tracks the existence of a line.

Parameters:

meta (LineMetaData) – Metadata about the line

Return type:

int

Returns:

the id of the registered line

register_predicate(meta)

Declare that a predicate exists.

Parameters:

meta (PredicateMetaData) – Metadata about the predicates

Return type:

int

Returns:

the id of the predicate, which can be used to identify the predicate during instrumentation.

reset()

Resets the subject properties.

Return type:

None

sharing_registries()

Create subject properties that share this instance’s registries.

The code object, predicate and line registries are filled during instrumentation, which happens once per module and is therefore global: every instrumented module reports the ids of these registries, no matter which executor triggered the execution. An executor that validates traces against its own, empty registries would reject every trace produced by instrumented code.

The tracer, in contrast, is per-execution. Auxiliary executors (assertion filtering, mutation analysis) therefore get a fresh tracer, so that they do not write into the tracer that the search uses, while still validating against the registries the instrumentation actually reports.

Return type:

SubjectProperties

Returns:

Subject properties sharing this instance’s registries, with a fresh tracer.

validate_execution_trace(execution_trace)

Validate the execution trace.

Parameters:

execution_trace (ExecutionTrace) – The execution trace to validate

Raises:

AssertionError – if the execution trace is invalid

Return type:

None

pynguin.ga.computations

Provides classes for computations on chromosomes, e.g., fitness and coverage.

class pynguin.ga.computations.BranchDistanceTestCaseFitnessFunction(executor, code_object_id)

A fitness function based on branch distances and entered code objects.

compute_fitness(individual)

Calculate the fitness value.

Parameters:

individual – the chromosome to compute the fitness for.

Returns:

DAR202

Return type:

the new fitness # noqa

compute_is_covered(individual)

Compute if the goal of this fitness function is covered.

This computation is usually cheaper than computing the fitness, because we are not interested in the distance, but only a boolean result.

Parameters:

individual – the chromosome to check coverage on.

Return type:

bool

Returns:

True, if the goal of this fitness function is covered.

is_maximisation_function()

Do we need to maximise or minimise this function?

Returns:

DAR202

Return type:

Whether or not this is a maximisation function # noqa

class pynguin.ga.computations.BranchDistanceTestSuiteFitnessFunction(executor)

A fitness function based on branch distances and entered code objects.

compute_fitness(individual)

Calculate the fitness value.

Parameters:

individual – the chromosome to compute the fitness for.

Returns:

DAR202

Return type:

the new fitness # noqa

compute_is_covered(individual)

Compute if the goal of this fitness function is covered.

This computation is usually cheaper than computing the fitness, because we are not interested in the distance, but only a boolean result.

Parameters:

individual – the chromosome to check coverage on.

Return type:

bool

Returns:

True, if the goal of this fitness function is covered.

is_maximisation_function()

Do we need to maximise or minimise this function?

Returns:

DAR202

Return type:

Whether or not this is a maximisation function # noqa

restrict(exclude_code, exclude_true, exclude_false)

Restrict this fitness function.

Restricts the fitness function with respect to the branches/code objects it considers.

Parameters:

• exclude_code (set[int]) – Ids of the code objects that should not be considered.

• exclude_true (set[int]) – Ids of predicates whose True branch should not be considered.

• exclude_false (set[int]) – Ids of predicates whose False branch should not be considered.

Return type:

None

class pynguin.ga.computations.ChromosomeComputation(_executor)

An abstract computation on chromosomes.

class pynguin.ga.computations.CoverageFunction

Interface for a coverage function.

abstract compute_coverage(individual)

Compute the coverage of the given individual.

Parameters:

individual – the chromosome to compute the coverage for.

Return type:

float

Returns:

The computed coverage.

class pynguin.ga.computations.FitnessFunction

Interface for a fitness function.

abstract compute_fitness(individual)

Calculate the fitness value.

Parameters:

individual – the chromosome to compute the fitness for.

Returns:

DAR202

Return type:

the new fitness # noqa

abstract compute_is_covered(individual)

Compute if the goal of this fitness function is covered.

This computation is usually cheaper than computing the fitness, because we are not interested in the distance, but only a boolean result.

Parameters:

individual – the chromosome to check coverage on.

Return type:

bool

Returns:

True, if the goal of this fitness function is covered.

abstract is_maximisation_function()

Do we need to maximise or minimise this function?

Returns:

DAR202

Return type:

Whether or not this is a maximisation function # noqa

class pynguin.ga.computations.LineTestSuiteFitnessFunction(_executor)

A fitness function based on lines covered and entered code objects.

compute_fitness(individual)

Calculate the fitness value.

Parameters:

individual – the chromosome to compute the fitness for.

Returns:

DAR202

Return type:

the new fitness # noqa

compute_is_covered(individual)

Compute if the goal of this fitness function is covered.

This computation is usually cheaper than computing the fitness, because we are not interested in the distance, but only a boolean result.

Parameters:

individual – the chromosome to check coverage on.

Return type:

bool

Returns:

True, if the goal of this fitness function is covered.

is_maximisation_function()

Do we need to maximise or minimise this function?

Returns:

DAR202

Return type:

Whether or not this is a maximisation function # noqa

class pynguin.ga.computations.StatementCheckedTestSuiteFitnessFunction(_executor)

A fitness function for the checked statement coverage of test suites.

A fitness function based on lines included in the backward slice of each statement of a test suite.

compute_fitness(individual)

Calculate the fitness value.

Parameters:

individual – the chromosome to compute the fitness for.

Returns:

DAR202

Return type:

the new fitness # noqa

compute_is_covered(individual)

Compute if the goal of this fitness function is covered.

This computation is usually cheaper than computing the fitness, because we are not interested in the distance, but only a boolean result.

Parameters:

individual – the chromosome to check coverage on.

Return type:

bool

Returns:

True, if the goal of this fitness function is covered.

is_maximisation_function()

Do we need to maximise or minimise this function?

Returns:

DAR202

Return type:

Whether or not this is a maximisation function # noqa

class pynguin.ga.computations.TestCaseAssertionCheckedCoverageFunction(_executor)

Computes checked coverage on test cases with assertions.

compute_coverage(individual)

Compute the coverage of the given individual.

Parameters:

individual – the chromosome to compute the coverage for.

Return type:

float

Returns:

The computed coverage.

class pynguin.ga.computations.TestCaseBranchCoverageFunction(_executor)

Computes branch coverage on test cases.

compute_coverage(individual)

Compute the coverage of the given individual.

Parameters:

individual – the chromosome to compute the coverage for.

Return type:

float

Returns:

The computed coverage.

class pynguin.ga.computations.TestCaseChromosomeComputation(_executor)

A function that computes something on a test case chromosome.

class pynguin.ga.computations.TestCaseCoverageFunction(_executor)

Base class for all coverage functions that act on test case level.

class pynguin.ga.computations.TestCaseFitnessFunction(executor, code_object_id)

Base class for test case fitness functions.

property code_object_id: int

The code object id, where the target of the fitness function is located.

Returns:

The code object id where the target of the fitness function is located.

class pynguin.ga.computations.TestCaseLineCoverageFunction(_executor)

Computes line coverage on test cases.

compute_coverage(individual)

Compute the coverage of the given individual.

Parameters:

individual – the chromosome to compute the coverage for.

Return type:

float

Returns:

The computed coverage.

class pynguin.ga.computations.TestCaseStatementCheckedCoverageFunction(_executor)

Computes checked coverage on the statements of test cases.

compute_coverage(individual)

Compute the coverage of the given individual.

Parameters:

individual – the chromosome to compute the coverage for.

Return type:

float

Returns:

The computed coverage.

class pynguin.ga.computations.TestSuiteAssertionCheckedCoverageFunction(_executor)

Computes checked coverage on test suites with assertions.

compute_coverage(individual)

Compute the coverage of the given individual.

Parameters:

individual – the chromosome to compute the coverage for.

Return type:

float

Returns:

The computed coverage.

class pynguin.ga.computations.TestSuiteBranchCoverageFunction(_executor)

Computes branch coverage on test suites.

compute_coverage(individual)

Compute the coverage of the given individual.

Parameters:

individual – the chromosome to compute the coverage for.

Return type:

float

Returns:

The computed coverage.

class pynguin.ga.computations.TestSuiteChromosomeComputation(_executor)

A function that computes something on a test suite chromosome.

class pynguin.ga.computations.TestSuiteCoverageFunction(_executor)

Base class for all coverage functions that act on test suite level.

class pynguin.ga.computations.TestSuiteFitnessFunction(_executor)

Base class for test suite fitness functions.

class pynguin.ga.computations.TestSuiteLineCoverageFunction(_executor)

Computes line coverage on test suites.

compute_coverage(individual)

Compute the coverage of the given individual.

Parameters:

individual – the chromosome to compute the coverage for.

Return type:

float

Returns:

The computed coverage.

class pynguin.ga.computations.TestSuiteStatementCheckedCoverageFunction(_executor)

Computes checked coverage on the statements of test suites.

compute_coverage(individual)

Compute the coverage of the given individual.

Parameters:

individual – the chromosome to compute the coverage for.

Return type:

float

Returns:

The computed coverage.

pynguin.assertion.assertiongenerator

Provides an assertion generator.

class pynguin.assertion.assertiongenerator.AssertionGenerator(plain_executor, filtering_executions=1, *, filtering_executor=None)

A simple assertion generator.

Creates all regression assertions.

visit_test_case_chromosome(chromosome)

Visit a test suite chromosome.

Parameters:

chromosome (TestCaseChromosome) – The test case chromosome

Return type:

None

visit_test_suite_chromosome(chromosome)

Visit a test suite chromosome.

Parameters:

chromosome (TestSuiteChromosome) – The test suite chromosome

Return type:

None

class pynguin.assertion.assertiongenerator.MutationAnalysisAssertionGenerator(plain_executor, mutation_controller, *, filtering_executor=None, testing=False)

Uses mutation analysis to filter out less relevant assertions.

pynguin.assertion.assertiongenerator.create_filtering_executor(plain_executor)

Build the executor to use for the assertion-filtering pass.

Returns a fresh-subprocess executor so that filtering catches per-process nondeterminism, or None (meaning: reuse the plain executor and filter in-process) when subprocess filtering is disabled or the plain executor already runs in a subprocess.

Parameters:

plain_executor (TestCaseExecutor) – The executor used for the in-process capture pass.

Return type:

TestCaseExecutor | None

Returns:

A subprocess executor for filtering, or None to filter in-process.

pynguin.assertion.mutation_analysis.controller

Provides a controller for generating mutants.

class pynguin.assertion.mutation_analysis.controller.MutationController(mutant_generator, module_ast, module)

A controller that creates mutants.

create_mutant(mutant_ast)

Creates a mutant of the module.

Parameters:

mutant_ast (Module) – The mutant AST.

Return type:

ModuleType

Returns:

The created mutant module.

create_mutants()

Creates mutants for the module.

Return type:

Generator[tuple[ModuleType | None, list[Mutation]]]

Returns:

A generator of tuples where the first entry is the mutated module or None if the mutated module cannot be created and the second part is a list of all the mutations operators applied.

mutant_count()

Calculates the number of mutants that can be created.

This is the pre-truncation total: if the mutant generator is configured to sample a subset (e.g. via a mutant-count cap), this still reports the full number of mutations the module yields.

Return type:

int

Returns:

The number of mutants that can be created.

pynguin.assertion.mutation_analysis.mutators

Provides classes for mutating ASTs.

Based on https://github.com/se2p/mutpy-pynguin/blob/main/mutpy/controller.py and integrated in Pynguin.

class pynguin.assertion.mutation_analysis.mutators.FirstOrderMutator(operators, *, maximum_mutants=-1, sampling_seed=0, reorder=False)

A mutator that applies first order mutations.

mutate(target_ast, module)

Mutate the given AST.

Parameters:

• target_ast (AST) – The AST to mutate.

• module (ModuleType) – The module to mutate.

Yields:

A generator of mutations and the mutated AST.

Return type:

Generator[tuple[list[Mutation], AST]]

mutation_count(target_ast, module)

Count the mutations the module yields before any truncation.

Parameters:

• target_ast (AST) – The AST to mutate.

• module (ModuleType) – The module to mutate.

Return type:

int

Returns:

The pre-truncation number of mutations.

class pynguin.assertion.mutation_analysis.mutators.HighOrderMutator(operators, hom_strategy=None)

A mutator that applies high order mutations.

mutate(target_ast, module)

Mutate the given AST.

Parameters:

• target_ast (AST) – The AST to mutate.

• module (ModuleType) – The module to mutate.

Yields:

A generator of mutations and the mutated AST.

Return type:

Generator[tuple[list[Mutation], AST]]

class pynguin.assertion.mutation_analysis.mutators.Mutator

A mutator is responsible for mutating an AST.

abstract mutate(target_ast, module)

Mutate the given AST.

Parameters:

• target_ast (AST) – The AST to mutate.

• module (ModuleType) – The module to mutate.

Yields:

A generator of mutations and the mutated AST.

Return type:

Generator[tuple[list[Mutation], AST]]

mutation_count(target_ast, module)

Count the mutations the module yields before any truncation.

Parameters:

• target_ast (AST) – The AST to mutate.

• module (ModuleType) – The module to mutate.

Return type:

int

Returns:

The pre-truncation number of mutations.

pynguin.instrumentation.version.DynamicSeedingInstrumentation

Specialized instrumentation adapter for dynamic constant seeding in Python 3.10.

pynguin.utils.typetracing

Provides utilities to trace the usage of objects.

class pynguin.utils.typetracing.DepthDefaultDict(depth)

A dictionary creating a UsageTraceNode automatically for each key.

The implementation creates a UsageTraceNode for each requested and non-existing key.

class pynguin.utils.typetracing.UsageTraceNode(name, depth=0, type_checks=<factory>, arg_types=<factory>, arg_values=<factory>)

The knowledge gathered by a proxy.

find_path(path)

Check if this usage trace tree has the given path.

Parameters:

path (tuple[str, ...]) – The path to check

Return type:

UsageTraceNode | None

Returns:

The usage trace node at the end of the path, if it exists, otherwise None.

static from_proxy(obj)

Extract knowledge from the given proxy.

This is a convenience method, because the knowledge attribute is not visible on a proxy.

Parameters:

obj (ObjectProxy) – the proxy from which we should extract knowledge

Return type:

UsageTraceNode

Returns:

The extracted knowledge.

merge(other)

Merge the knowledge from the other proxy into this one.

Parameters:

other (UsageTraceNode) – The knowledge that should be merged into this one.

Return type:

None

pretty()

Create a pretty representation of this object.

Return type:

str

Returns:

A nicely formatted string

pynguin.utils.typetracing.proxify(*, log_args=False, no_wrap_return=False)

Decorator method to trace the usage of a method on a proxy.

1. Unwraps the proxy.

2. Stores the access to the method

3. Stores the argument types if requested.

4. Stores the argument values if requested and if the type is in VALUE_TRACED_TYPES.

5. Wraps the result in a proxy object (unless requested otherwise).

Parameters:

• log_args – Should we store the arguments (types and values)?

• no_wrap_return – Some cases, e.g., __int__ don’t allow a return value that is not an int, so in some cases we have to disable wrapping.

Returns:

A decorated function

pynguin.utils.typetracing.shim_isinstance()

Context manager that temporarily replaces isinstance with a shim.

The shim is aware of ObjectProxies.

Yields:

resets the shim

pynguin.utils.typetracing.unwrap(obj)

Unwrap the given object if it is a Proxy.

Parameters:

obj – The object to unwrap

Returns:

The unwrapped object