parallel_execution module¶
Parallel execution of disciplines and functions using multiprocessing.
- class gemseo.core.parallel_execution.DiscParallelExecution(workers, n_processes=2, use_threading=False, wait_time_between_fork=0.0, exceptions_to_re_raise=None)[source]¶
Bases:
gemseo.core.parallel_execution.ParallelExecution
Execute disciplines in parallel.
- Parameters
workers (ParallelExecutionWorkerType) – The objects that perform the tasks. Either pass one worker, and it will be forked in multiprocessing. Or, when using multithreading or different workers, pass one worker per input data.
n_processes (int) –
The maximum simultaneous number of threads, if
use_threading
is True, or processes otherwise, used to parallelize the execution.By default it is set to 2.
use_threading (bool) –
If True, use threads instead of processes to parallelize the execution. Multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory. This is important to note if you want to execute the same discipline multiple times, in which case you shall use multiprocessing.
By default it is set to False.
wait_time_between_fork (float) –
The time to wait between two forks of the process/thread.
By default it is set to 0.0.
exceptions_to_re_raise (tuple[type[Exception]] | None) –
The exceptions that should be raised again when caught inside a worker. If None, all exceptions coming from workers are caught and the execution is allowed to continue.
By default it is set to None.
- Raises
ValueError – If there are duplicated workers in workers when using multithreading.
- Return type
None
- execute(input_values, exec_callback=None, task_submitted_callback=None)¶
Execute all the processes.
- Parameters
input_values (Sequence[ndarray] | ndarray) – The input values.
exec_callback (Callable[[int, Any], Any] | None) –
A callback function called with the pair (index, outputs) as arguments when an item is retrieved from the processing. Index is the associated index in input_values of the input used to compute the outputs. If None, no function is called.
By default it is set to None.
task_submitted_callback (Callable | None) –
A callback function called when all the tasks are submitted, but not done yet. If None, no function is called.
By default it is set to None.
- Returns
The computed outputs.
- Raises
TypeError – If the exec_callback is not callable. If the task_submitted_callback is not callable.
- Return type
Warning
This class relies on multiprocessing features, it is therefore necessary to protect its execution with an
if __name__ == '__main__':
statement when working on Windows.
- N_CPUS = 2¶
- workers: ParallelExecutionWorkerType¶
The objects that perform the tasks.
- class gemseo.core.parallel_execution.DiscParallelLinearization(workers, n_processes=2, use_threading=False, wait_time_between_fork=0.0, exceptions_to_re_raise=None)[source]¶
Bases:
gemseo.core.parallel_execution.ParallelExecution
Linearize disciplines in parallel.
- Parameters
workers (ParallelExecutionWorkerType) – The objects that perform the tasks. Either pass one worker, and it will be forked in multiprocessing. Or, when using multithreading or different workers, pass one worker per input data.
n_processes (int) –
The maximum simultaneous number of threads, if
use_threading
is True, or processes otherwise, used to parallelize the execution.By default it is set to 2.
use_threading (bool) –
If True, use threads instead of processes to parallelize the execution. Multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory. This is important to note if you want to execute the same discipline multiple times, in which case you shall use multiprocessing.
By default it is set to False.
wait_time_between_fork (float) –
The time to wait between two forks of the process/thread.
By default it is set to 0.0.
exceptions_to_re_raise (tuple[type[Exception]] | None) –
The exceptions that should be raised again when caught inside a worker. If None, all exceptions coming from workers are caught and the execution is allowed to continue.
By default it is set to None.
- Raises
ValueError – If there are duplicated workers in workers when using multithreading.
- Return type
None
- execute(input_values, exec_callback=None, task_submitted_callback=None)¶
Execute all the processes.
- Parameters
input_values (Sequence[ndarray] | ndarray) – The input values.
exec_callback (Callable[[int, Any], Any] | None) –
A callback function called with the pair (index, outputs) as arguments when an item is retrieved from the processing. Index is the associated index in input_values of the input used to compute the outputs. If None, no function is called.
By default it is set to None.
task_submitted_callback (Callable | None) –
A callback function called when all the tasks are submitted, but not done yet. If None, no function is called.
By default it is set to None.
- Returns
The computed outputs.
- Raises
TypeError – If the exec_callback is not callable. If the task_submitted_callback is not callable.
- Return type
Warning
This class relies on multiprocessing features, it is therefore necessary to protect its execution with an
if __name__ == '__main__':
statement when working on Windows.
- N_CPUS = 2¶
- workers: ParallelExecutionWorkerType¶
The objects that perform the tasks.
- class gemseo.core.parallel_execution.ParallelExecution(workers, n_processes=2, use_threading=False, wait_time_between_fork=0.0, exceptions_to_re_raise=None)[source]¶
Bases:
object
Perform a parallel execution of tasks on input values.
Input values must be a list of independent pointers.
- Parameters
workers (ParallelExecutionWorkerType) – The objects that perform the tasks. Either pass one worker, and it will be forked in multiprocessing. Or, when using multithreading or different workers, pass one worker per input data.
n_processes (int) –
The maximum simultaneous number of threads, if
use_threading
is True, or processes otherwise, used to parallelize the execution.By default it is set to 2.
use_threading (bool) –
If True, use threads instead of processes to parallelize the execution. Multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory. This is important to note if you want to execute the same discipline multiple times, in which case you shall use multiprocessing.
By default it is set to False.
wait_time_between_fork (float) –
The time to wait between two forks of the process/thread.
By default it is set to 0.0.
exceptions_to_re_raise (tuple[type[Exception]] | None) –
The exceptions that should be raised again when caught inside a worker. If None, all exceptions coming from workers are caught and the execution is allowed to continue.
By default it is set to None.
- Raises
ValueError – If there are duplicated workers in workers when using multithreading.
- Return type
None
- execute(input_values, exec_callback=None, task_submitted_callback=None)[source]¶
Execute all the processes.
- Parameters
input_values (Sequence[ndarray] | ndarray) – The input values.
exec_callback (Callable[[int, Any], Any] | None) –
A callback function called with the pair (index, outputs) as arguments when an item is retrieved from the processing. Index is the associated index in input_values of the input used to compute the outputs. If None, no function is called.
By default it is set to None.
task_submitted_callback (Callable | None) –
A callback function called when all the tasks are submitted, but not done yet. If None, no function is called.
By default it is set to None.
- Returns
The computed outputs.
- Raises
TypeError – If the exec_callback is not callable. If the task_submitted_callback is not callable.
- Return type
Warning
This class relies on multiprocessing features, it is therefore necessary to protect its execution with an
if __name__ == '__main__':
statement when working on Windows.
- N_CPUS = 2¶
- workers: ParallelExecutionWorkerType¶
The objects that perform the tasks.
- gemseo.core.parallel_execution.worker(par_exe, queue_in, queue_out)[source]¶
Execute a function while there are args left in the queue_in.
- Parameters
par_exe (ParallelExecution | DiscParallelExecution | DiscParallelLinearization) – The parallel execution object that contains the function to be executed.
queue_in (queue.Queue) – The inputs to be evaluated.
queue_out (queue.Queue) – The queue object where the outputs of the function will be saved.
- Return type
None