gemseo / core

parallel_execution module

Parallel execution of disciplines and functions using multiprocessing.

Classes:

DiscParallelExecution(worker_list[, ...])

Execute disciplines in parallel.

DiscParallelLinearization(worker_list[, ...])

Linearize disciplines in parallel.

ParallelExecution(worker_list[, ...])

Perform a parallel execution of tasks on input values.

Functions:

worker(par_exe, queue_in, queue_out)

Execute a function while there are args left in the queue_in.
class gemseo.core.parallel_execution.DiscParallelExecution(worker_list, n_processes=2, use_threading=False, wait_time_between_fork=0.0)[source]

Bases: gemseo.core.parallel_execution.ParallelExecution

Execute disciplines in parallel.

Parameters

  • worker_list (ParallelExecutionWorkerType) – The objects that perform the tasks. Either pass one worker, and it will be forked in multiprocessing. Or, when using multithreading or different workers, pass one worker per input data.

  • n_processes (int) –

    The maximum number of processes on which to run.

    By default it is set to 2.

  • use_threading (bool) –

    If True, use Threads instead of processes to parallelize the execution. Multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory. This is important to note if you want to execute the same discipline multiple times, in which case you shall use multiprocessing.

    By default it is set to False.

  • wait_time_between_fork (float) –

    The time to wait between two forks of the process/Thread.

    By default it is set to 0.0.

Raises

ValueError – If there are duplicated workers in worker_list when using multithreading.

Return type

None

Attributes:

N_CPUS

Methods:

execute(input_data_list[, exec_callback, ...])

Execute all the processes.
N_CPUS = 2
execute(input_data_list, exec_callback=None, task_submitted_callback=None)

Execute all the processes.

Parameters

  • input_data_list (Union[Sequence[numpy.ndarray], numpy.ndarray]) – The input values.

  • exec_callback (Optional[Callable[[int, Any], Any]]) –

    A callback function called with the pair (index, outputs) as arguments when an item is retrieved from the processing. Index is the associated index in input_data_list of the input used to compute the outputs. If None, no function is called.

    By default it is set to None.

  • task_submitted_callback (Optional[Callable]) –

    A callback function called when all the tasks are submitted, but not done yet. If None, no function is called.

    By default it is set to None.

Returns

The computed outputs.

Raises

TypeError – If the exec_callback is not callable. If the task_submitted_callback is not callable.

Return type

Dict[int, Any]

class gemseo.core.parallel_execution.DiscParallelLinearization(worker_list, n_processes=2, use_threading=False, wait_time_between_fork=0.0)[source]

Bases: gemseo.core.parallel_execution.ParallelExecution

Linearize disciplines in parallel.

Parameters

  • worker_list (ParallelExecutionWorkerType) – The objects that perform the tasks. Either pass one worker, and it will be forked in multiprocessing. Or, when using multithreading or different workers, pass one worker per input data.

  • n_processes (int) –

    The maximum number of processes on which to run.

    By default it is set to 2.

  • use_threading (bool) –

    If True, use Threads instead of processes to parallelize the execution. Multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory. This is important to note if you want to execute the same discipline multiple times, in which case you shall use multiprocessing.

    By default it is set to False.

  • wait_time_between_fork (float) –

    The time to wait between two forks of the process/Thread.

    By default it is set to 0.0.

Raises

ValueError – If there are duplicated workers in worker_list when using multithreading.

Return type

None

Attributes:

N_CPUS

Methods:

execute(input_data_list[, exec_callback, ...])

Execute all the processes.
N_CPUS = 2
execute(input_data_list, exec_callback=None, task_submitted_callback=None)

Execute all the processes.

Parameters

  • input_data_list (Union[Sequence[numpy.ndarray], numpy.ndarray]) – The input values.

  • exec_callback (Optional[Callable[[int, Any], Any]]) –

    A callback function called with the pair (index, outputs) as arguments when an item is retrieved from the processing. Index is the associated index in input_data_list of the input used to compute the outputs. If None, no function is called.

    By default it is set to None.

  • task_submitted_callback (Optional[Callable]) –

    A callback function called when all the tasks are submitted, but not done yet. If None, no function is called.

    By default it is set to None.

Returns

The computed outputs.

Raises

TypeError – If the exec_callback is not callable. If the task_submitted_callback is not callable.

Return type

Dict[int, Any]

class gemseo.core.parallel_execution.ParallelExecution(worker_list, n_processes=2, use_threading=False, wait_time_between_fork=0.0)[source]

Bases: object

Perform a parallel execution of tasks on input values.

Input values must be a list of independent pointers.

Parameters

  • worker_list (ParallelExecutionWorkerType) – The objects that perform the tasks. Either pass one worker, and it will be forked in multiprocessing. Or, when using multithreading or different workers, pass one worker per input data.

  • n_processes (int) –

    The maximum number of processes on which to run.

    By default it is set to 2.

  • use_threading (bool) –

    If True, use Threads instead of processes to parallelize the execution. Multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory. This is important to note if you want to execute the same discipline multiple times, in which case you shall use multiprocessing.

    By default it is set to False.

  • wait_time_between_fork (float) –

    The time to wait between two forks of the process/Thread.

    By default it is set to 0.0.

Raises

ValueError – If there are duplicated workers in worker_list when using multithreading.

Return type

None

Attributes:

N_CPUS

Methods:

execute(input_data_list[, exec_callback, ...])

Execute all the processes.
N_CPUS = 2
execute(input_data_list, exec_callback=None, task_submitted_callback=None)[source]

Execute all the processes.

Parameters

  • input_data_list (Union[Sequence[numpy.ndarray], numpy.ndarray]) – The input values.

  • exec_callback (Optional[Callable[[int, Any], Any]]) –

    A callback function called with the pair (index, outputs) as arguments when an item is retrieved from the processing. Index is the associated index in input_data_list of the input used to compute the outputs. If None, no function is called.

    By default it is set to None.

  • task_submitted_callback (Optional[Callable]) –

    A callback function called when all the tasks are submitted, but not done yet. If None, no function is called.

    By default it is set to None.

Returns

The computed outputs.

Raises

TypeError – If the exec_callback is not callable. If the task_submitted_callback is not callable.

Return type

Dict[int, Any]

gemseo.core.parallel_execution.worker(par_exe, queue_in, queue_out)[source]

Execute a function while there are args left in the queue_in.

Parameters
Return type

None