Calibrate or select a machine learning algorithm¶

Calibration of a machine learning algorithm.

A machine learning algorithm depends on hyper-parameters, e.g. the number of clusters for a clustering algorithm, the regularization constant for a regression model, the kernel for a Gaussian process regression, … Its ability to generalize the information learned during the training stage, and thus to avoid over-fitting, which is an over-reliance on the learning data set, depends on the values of these hyper-parameters. Thus, the hyper-parameters minimizing the learning quality measure are rarely those minimizing the generalization one. Classically, the generalization one decreases before growing again as the model becomes more complex, while the learning error keeps decreasing. This phenomenon is called the curse of dimensionality.

In this module, the MLAlgoCalibration class aims to calibrate the hyper-parameters in order to minimize this measure of the generalization quality over a calibration parameter space. This class relies on the MLAlgoAssessor class which is a discipline (MDODiscipline) built from a machine learning algorithm (MLAlgo), a dataset (Dataset), a quality measure (MLQualityMeasure) and various options for the data scaling, the quality measure and the machine learning algorithm. The inputs of this discipline are hyper-parameters of the machine learning algorithm while the output is the quality criterion.

class gemseo.mlearning.core.calibration.MLAlgoAssessor(algo, dataset, parameters, measure, measure_options=None, transformer=None, **algo_options)[source]

Discipline assessing the quality of a machine learning algorithm.

This quality depends on the values of parameters to calibrate with the MLAlgoCalibration.

Parameters

algo (str) – The name of a machine learning algorithm.
dataset (Dataset) – A learning dataset.
parameters (Iterable[str]) – The parameters of the machine learning algorithm to calibrate.
measure (type[MLQualityMeasure]) – A measure to assess the machine learning algorithm.
measure_options (MeasureOptionsType | None) –
The options of the quality measure. If “multioutput” is missing, it is added with False as value. If None, do not use quality measure options.

By default it is set to None.
transformer (Mapping[str, TransformerType] | None) –
The strategies to transform the variables. The values are instances of Transformer while the keys are the names of either the variables or the groups of variables, e.g. “inputs” or “outputs” in the case of the regression algorithms. If a group is specified, the Transformer will be applied to all the variables of this group. If None, do not transform the variables.

By default it is set to None.
**algo_options (MLAlgoParameterType) – The options of the machine learning algorithm.

Raises

ValueError – If the measure option “multioutput” is True.

Return type

None

classmethod activate_time_stamps()

Activate the time stamps.

For storing start and end times of execution and linearizations.

Return type: None

add_differentiated_inputs(inputs=None)

Add inputs against which to differentiate the outputs.

This method updates MDODiscipline._differentiated_inputs with inputs.

Parameters

inputs (Iterable[str] | None) –

The input variables against which to differentiate the outputs. If None, all the inputs of the discipline are used.

By default it is set to None.

Raises

ValueError – When the inputs wrt which differentiate the discipline are not inputs of the latter.

Return type

None

add_differentiated_outputs(outputs=None)

Add outputs to be differentiated.

This method updates MDODiscipline._differentiated_outputs with outputs.

Parameters

outputs (Iterable[str] | None) –

The output variables to be differentiated. If None, all the outputs of the discipline are used.

By default it is set to None.

Raises

ValueError – When the outputs to differentiate are not discipline outputs.

Return type

None

add_namespace_to_input(name, namespace)

Add a namespace prefix to an existing input grammar element.

The updated input grammar element name will be namespace``+:data:`~gemseo.core.namespaces.namespace_separator`+``name.

Parameters

name (str) – The element name to rename.
namespace (str) – The name of the namespace.

add_namespace_to_output(name, namespace)

Add a namespace prefix to an existing output grammar element.

The updated output grammar element name will be namespace``+:data:`~gemseo.core.namespaces.namespace_separator`+``name.

Parameters

name (str) – The element name to rename.
namespace (str) – The name of the namespace.

add_status_observer(obs)

Add an observer for the status.

Add an observer for the status to be notified when self changes of status.

Parameters: obs (Any) – The observer to add.
Return type: None

auto_get_grammar_file(is_input=True, name=None, comp_dir=None)

Use a naming convention to associate a grammar file to the discipline.

Search in the directory comp_dir for either an input grammar file named name + "_input.json" or an output grammar file named name + "_output.json".

Parameters

is_input (bool) –
Whether to search for an input or output grammar file.

By default it is set to True.
name (str | None) –
The name to be searched in the file names. If None, use the name of the discipline class.

By default it is set to None.
comp_dir (str | Path | None) –
The directory in which to search the grammar file. If None, use the GRAMMAR_DIRECTORY if any, or the directory of the discipline class module.

By default it is set to None.

Returns

The grammar file path.

Return type

str

check_input_data(input_data, raise_exception=True)

Check the input data validity.

Parameters

input_data (dict[str, Any]) – The input data needed to execute the discipline according to the discipline input grammar.
raise_exception (bool) –
Whether to raise on error.

By default it is set to True.

Return type

None

check_jacobian(input_data=None, derr_approx='finite_differences', step=1e-07, threshold=1e-08, linearization_mode='auto', inputs=None, outputs=None, parallel=False, n_processes=2, use_threading=False, wait_time_between_fork=0, auto_set_step=False, plot_result=False, file_path='jacobian_errors.pdf', show=False, fig_size_x=10, fig_size_y=10, reference_jacobian_path=None, save_reference_jacobian=False, indices=None)

Check if the analytical Jacobian is correct with respect to a reference one.

If reference_jacobian_path is not None and save_reference_jacobian is True, compute the reference Jacobian with the approximation method and save it in reference_jacobian_path.

If reference_jacobian_path is not None and save_reference_jacobian is False, do not compute the reference Jacobian but read it from reference_jacobian_path.

If reference_jacobian_path is None, compute the reference Jacobian without saving it.

Parameters

input_data (dict[str, ndarray] | None) –
The input data needed to execute the discipline according to the discipline input grammar. If None, use the MDODiscipline.default_inputs.

By default it is set to None.
derr_approx (str) –
The approximation method, either “complex_step” or “finite_differences”.

By default it is set to finite_differences.
threshold (float) –
The acceptance threshold for the Jacobian error.

By default it is set to 1e-08.
linearization_mode (str) –
the mode of linearization: direct, adjoint or automated switch depending on dimensions of inputs and outputs (Default value = ‘auto’)

By default it is set to auto.
inputs (Iterable[str] | None) –
The names of the inputs wrt which to differentiate the outputs.

By default it is set to None.
outputs (Iterable[str] | None) –
The names of the outputs to be differentiated.

By default it is set to None.
step (float) –
The differentiation step.

By default it is set to 1e-07.
parallel (bool) –
Whether to differentiate the discipline in parallel.

By default it is set to False.
n_processes (int) –
The maximum simultaneous number of threads, if use_threading is True, or processes otherwise, used to parallelize the execution.

By default it is set to 2.
use_threading (bool) –
Whether to use threads instead of processes to parallelize the execution; multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory This is important to note if you want to execute the same discipline multiple times, you shall use multiprocessing.

By default it is set to False.
wait_time_between_fork (float) –
The time waited between two forks of the process / thread.

By default it is set to 0.
auto_set_step (bool) –
Whether to compute the optimal step for a forward first order finite differences gradient approximation.

By default it is set to False.
plot_result (bool) –
Whether to plot the result of the validation (computed vs approximated Jacobians).

By default it is set to False.
file_path (str | Path) –
The path to the output file if plot_result is True.

By default it is set to jacobian_errors.pdf.
show (bool) –
Whether to open the figure.

By default it is set to False.
fig_size_x (float) –
The x-size of the figure in inches.

By default it is set to 10.
fig_size_y (float) –
The y-size of the figure in inches.

By default it is set to 10.
reference_jacobian_path (str | Path | None) –
The path of the reference Jacobian file.

By default it is set to None.
save_reference_jacobian (bool) –
Whether to save the reference Jacobian.

By default it is set to False.
indices (Iterable[int] | None) –
The indices of the inputs and outputs for the different sub-Jacobian matrices, formatted as {variable_name: variable_components} where variable_components can be either an integer, e.g. 2 a sequence of integers, e.g. [0, 3], a slice, e.g. slice(0,3), the ellipsis symbol (…) or None, which is the same as ellipsis. If a variable name is missing, consider all its components. If None, consider all the components of all the inputs and outputs.

By default it is set to None.

Returns

Whether the analytical Jacobian is correct with respect to the reference one.

check_output_data(raise_exception=True)

Check the output data validity.

Parameters

raise_exception (bool) –

Whether to raise an exception when the data is invalid.

By default it is set to True.

Return type

None

classmethod deactivate_time_stamps()

Deactivate the time stamps.

For storing start and end times of execution and linearizations.

Return type: None

static deserialize(file_path)

Deserialize a discipline from a file.

Parameters: file_path (str | Path) – The path to the file containing the discipline.
Returns: The discipline instance.
Return type: MDODiscipline

execute(input_data=None)

Execute the discipline.

This method executes the discipline:

Adds the default inputs to the input_data if some inputs are not defined in input_data but exist in MDODiscipline.default_inputs.
Checks whether the last execution of the discipline was called with identical inputs, i.e. cached in MDODiscipline.cache; if so, directly returns self.cache.get_output_cache(inputs).
Caches the inputs.
Checks the input data against MDODiscipline.input_grammar.
If MDODiscipline.data_processor is not None, runs the preprocessor.
Updates the status to MDODiscipline.STATUS_RUNNING.
Calls the MDODiscipline._run() method, that shall be defined.
If MDODiscipline.data_processor is not None, runs the postprocessor.
Checks the output data.
Caches the outputs.
Updates the status to MDODiscipline.STATUS_DONE or MDODiscipline.STATUS_FAILED.
Updates summed execution time.

Parameters

input_data (Mapping[str, Any] | None) –

The input data needed to execute the discipline according to the discipline input grammar. If None, use the MDODiscipline.default_inputs.

By default it is set to None.

Returns

The discipline local data after execution.

Raises

RuntimeError – When residual_variables are declared but self.run_solves_residuals is False. This is not suported yet.

Return type

dict[str, Any]

get_all_inputs()

Return the local input data as a list.

The order is given by MDODiscipline.get_input_data_names().

Returns: The local input data.
Return type: list[Any]

get_all_outputs()

Return the local output data as a list.

The order is given by MDODiscipline.get_output_data_names().

Returns: The local output data.
Return type: list[Any]

get_attributes_to_serialize()

Define the names of the attributes to be serialized.

Shall be overloaded by disciplines

Returns: The names of the attributes to be serialized.
Return type: list[str]

static get_data_list_from_dict(keys, data_dict)

Filter the dict from a list of keys or a single key.

If keys is a string, then the method return the value associated to the key. If keys is a list of strings, then the method returns a generator of value corresponding to the keys which can be iterated.

Parameters

keys (str | Iterable) – One or several names.
data_dict (dict[str, Any]) – The mapping from which to get the data.

Returns

Either a data or a generator of data.

Return type

Any | Generator[Any]

get_disciplines_in_dataflow_chain()

Return the disciplines that must be shown as blocks within the XDSM representation of a chain.

By default, only the discipline itself is shown. This function can be differently implemented for any type of inherited discipline.

Returns: The disciplines shown in the XDSM chain.
Return type: list[gemseo.core.discipline.MDODiscipline]

get_expected_dataflow()

Return the expected data exchange sequence.

This method is used for the XDSM representation.

The default expected data exchange sequence is an empty list.

See also

MDOFormulation.get_expected_dataflow

Returns: The data exchange arcs.
Return type: list[tuple[gemseo.core.discipline.MDODiscipline, gemseo.core.discipline.MDODiscipline, list[str]]]

get_expected_workflow()

Return the expected execution sequence.

This method is used for the XDSM representation.

The default expected execution sequence is the execution of the discipline itself.

See also

MDOFormulation.get_expected_workflow

Returns: The expected execution sequence.
Return type: SerialExecSequence

get_input_data(with_namespaces=True)

Return the local input data as a dictionary.

Parameters

with_namespaces –

Whether to keep the namespace prefix of the input names, if any.

By default it is set to True.

Returns

The local input data.

Return type

dict[str, Any]

get_input_data_names(with_namespaces=True)

Return the names of the input variables.

Parameters

with_namespaces –

Whether to keep the namespace prefix of the input names, if any.

By default it is set to True.

Returns

The names of the input variables.

Return type

list[str]

get_input_output_data_names(with_namespaces=True)

Return the names of the input and output variables.

Args:

with_namespaces: Whether to keep the namespace prefix of the
output names, if any.

Returns: The name of the input and output variables.
Return type: list[str]

get_inputs_asarray()

Return the local output data as a large NumPy array.

The order is the one of MDODiscipline.get_all_outputs().

Returns: The local output data.
Return type: numpy.ndarray

get_inputs_by_name(data_names)

Return the local data associated with input variables.

Parameters: data_names (Iterable[str]) – The names of the input variables.
Returns: The local data for the given input variables.
Raises: ValueError – When a variable is not an input of the discipline.
Return type: list[Any]

get_local_data_by_name(data_names)

Return the local data of the discipline associated with variables names.

Parameters: data_names (Iterable[str]) – The names of the variables.
Returns: The local data associated with the variables names.
Raises: ValueError – When a name is not a discipline input name.
Return type: Generator[Any]

get_output_data(with_namespaces=True)

Return the local output data as a dictionary.

Parameters

with_namespaces –

Whether to keep the namespace prefix of the output names, if any.

By default it is set to True.

Returns

The local output data.

Return type

dict[str, Any]

get_output_data_names(with_namespaces=True)

Return the names of the output variables.

Parameters

with_namespaces –

Whether to keep the namespace prefix of the output names, if any.

By default it is set to True.

Returns

The names of the output variables.

Return type

list[str]

get_outputs_asarray()

Return the local input data as a large NumPy array.

The order is the one of MDODiscipline.get_all_inputs().

Returns: The local input data.
Return type: numpy.ndarray

get_outputs_by_name(data_names)

Return the local data associated with output variables.

Parameters: data_names (Iterable[str]) – The names of the output variables.
Returns: The local data for the given output variables.
Raises: ValueError – When a variable is not an output of the discipline.
Return type: list[Any]

get_sub_disciplines()

Return the sub-disciplines if any.

Returns: The sub-disciplines.
Return type: list[gemseo.core.discipline.MDODiscipline]

is_all_inputs_existing(data_names)

Test if several variables are discipline inputs.

Parameters: data_names (Iterable[str]) – The names of the variables.
Returns: Whether all the variables are discipline inputs.
Return type: bool

is_all_outputs_existing(data_names)

Test if several variables are discipline outputs.

Parameters: data_names (Iterable[str]) – The names of the variables.
Returns: Whether all the variables are discipline outputs.
Return type: bool

is_input_existing(data_name)

Test if a variable is a discipline input.

Parameters: data_name (str) – The name of the variable.
Returns: Whether the variable is a discipline input.
Return type: bool

is_output_existing(data_name)

Test if a variable is a discipline output.

Parameters: data_name (str) – The name of the variable.
Returns: Whether the variable is a discipline output.
Return type: bool

static is_scenario()

Whether the discipline is a scenario.

Return type: bool

linearize(input_data=None, force_all=False, force_no_exec=False)

Execute the linearized version of the code.

Parameters

input_data (dict[str, Any] | None) –
The input data needed to linearize the discipline according to the discipline input grammar. If None, use the MDODiscipline.default_inputs.

By default it is set to None.
force_all (bool) –
If False, MDODiscipline._differentiated_inputs and MDODiscipline._differentiated_outputs are used to filter the differentiated variables. otherwise, all outputs are differentiated wrt all inputs.

By default it is set to False.
force_no_exec (bool) –
If True, the discipline is not re-executed, cache is loaded anyway.

By default it is set to False.

Returns

The Jacobian of the discipline.

Return type

dict[str, dict[str, ndarray]]

notify_status_observers()

Notify all status observers that the status has changed.

Return type: None

remove_status_observer(obs)

Remove an observer for the status.

Parameters: obs (Any) – The observer to remove.
Return type: None

reset_statuses_for_run()

Set all the statuses to MDODiscipline.STATUS_PENDING.

Raises: ValueError – When the discipline cannot be run because of its status.
Return type: None

serialize(file_path)

Serialize the discipline and store it in a file.

Parameters: file_path (str | Path) – The path to the file to store the discipline.
Return type: None

set_cache_policy(cache_type='SimpleCache', cache_tolerance=0.0, cache_hdf_file=None, cache_hdf_node_name=None, is_memory_shared=True)

Set the type of cache to use and the tolerance level.

This method defines when the output data have to be cached according to the distance between the corresponding input data and the input data already cached for which output data are also cached.

The cache can be either a SimpleCache recording the last execution or a cache storing all executions, e.g. MemoryFullCache and HDF5Cache. Caching data can be either in-memory, e.g. SimpleCache and MemoryFullCache, or on the disk, e.g. HDF5Cache.

The attribute CacheFactory.caches provides the available caches types.

Parameters

cache_type (str) –
The type of cache.

By default it is set to SimpleCache.
cache_tolerance (float) –
The maximum relative norm of the difference between two input arrays to consider that two input arrays are equal.

By default it is set to 0.0.
cache_hdf_file (str | Path | None) –
The path to the HDF file to store the data; this argument is mandatory when the MDODiscipline.HDF5_CACHE policy is used.

By default it is set to None.
cache_hdf_node_name (str | None) –
The name of the HDF file node to store the discipline data. If None, MDODiscipline.name is used.

By default it is set to None.
is_memory_shared (bool) –
Whether to store the data with a shared memory dictionary, which makes the cache compatible with multiprocessing.

By default it is set to True.

Return type

None

set_disciplines_statuses(status)

Set the sub-disciplines statuses.

To be implemented in subclasses.

Parameters: status (str) – The status.
Return type: None

set_jacobian_approximation(jac_approx_type='finite_differences', jax_approx_step=1e-07, jac_approx_n_processes=1, jac_approx_use_threading=False, jac_approx_wait_time=0)

Set the Jacobian approximation method.

Sets the linearization mode to approx_method, sets the parameters of the approximation for further use when calling MDODiscipline.linearize().

Parameters

jac_approx_type (str) –
The approximation method, either “complex_step” or “finite_differences”.

By default it is set to finite_differences.
jax_approx_step (float) –
The differentiation step.

By default it is set to 1e-07.
jac_approx_n_processes (int) –
The maximum simultaneous number of threads, if jac_approx_use_threading is True, or processes otherwise, used to parallelize the execution.

By default it is set to 1.
jac_approx_use_threading (bool) –
Whether to use threads instead of processes to parallelize the execution; multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory This is important to note if you want to execute the same discipline multiple times, you shall use multiprocessing.

By default it is set to False.
jac_approx_wait_time (float) –
The time waited between two forks of the process / thread.

By default it is set to 0.

Return type

None

set_optimal_fd_step(outputs=None, inputs=None, force_all=False, print_errors=False, numerical_error=2.220446049250313e-16)

Compute the optimal finite-difference step.

Compute the optimal step for a forward first order finite differences gradient approximation. Requires a first evaluation of the perturbed functions values. The optimal step is reached when the truncation error (cut in the Taylor development), and the numerical cancellation errors (round-off when doing f(x+step)-f(x)) are approximately equal.

Warning

This calls the discipline execution twice per input variables.

See also

https://en.wikipedia.org/wiki/Numerical_differentiation and “Numerical Algorithms and Digital Representation”, Knut Morken , Chapter 11, “Numerical Differentiation”

Parameters

inputs (Iterable[str] | None) –
The inputs wrt which the outputs are linearized. If None, use the MDODiscipline._differentiated_inputs.

By default it is set to None.
outputs (Iterable[str] | None) –
The outputs to be linearized. If None, use the MDODiscipline._differentiated_outputs.

By default it is set to None.
force_all (bool) –
Whether to consider all the inputs and outputs of the discipline;

By default it is set to False.
print_errors (bool) –
Whether to display the estimated errors.

By default it is set to False.
numerical_error (float) –
The numerical error associated to the calculation of f. By default, this is the machine epsilon (appx 1e-16), but can be higher when the calculation of f requires a numerical resolution.

By default it is set to 2.220446049250313e-16.

Returns

The estimated errors of truncation and cancellation error.

Raises

ValueError – When the Jacobian approximation method has not been set.

store_local_data(**kwargs)

Store discipline data in local data.

Parameters: **kwargs (Any) – The data to be stored in MDODiscipline.local_data.
Return type: None

algo: str: The name of a machine learning algorithm.

algos: list[MLAlgo]: The instances of the machine learning algorithm (one per execution of the machine learning algorithm assessor).

property cache_tol: float

The cache input tolerance.

This is the tolerance for equality of the inputs in the cache. If norm(stored_input_data-input_data) <= cache_tol * norm(stored_input_data), the cached data for stored_input_data is returned when calling self.execute(input_data).

Raises: ValueError – When the discipline does not have a cache.

dataset: Dataset: The learning dataset.

property default_inputs: dict[str, Any]

The default inputs.

Raises: TypeError – When the default inputs are not passed as a dictionary.

property exec_time: float | None

The cumulated execution time of the discipline.

This property is multiprocessing safe.

Raises: RuntimeError – When the discipline counters are disabled.

property grammar_type: gemseo.core.grammars.base_grammar.BaseGrammar: The type of grammar to be used for inputs and outputs declaration.

property linearization_mode: str

The linearization mode among MDODiscipline.AVAILABLE_MODES.

Raises: ValueError – When the linearization mode is unknown.

property local_data: gemseo.core.discipline_data.DisciplineData: The current input and output data.

measure: MLQualityMeasure: The measure to assess the machine learning algorithm.

measure_options: dict[str, int | Dataset]: The options of the quality measure.

property n_calls: int | None

The number of times the discipline was executed.

This property is multiprocessing safe.

Raises: RuntimeError – When the discipline counters are disabled.

property n_calls_linearize: int | None

The number of times the discipline was linearized.

This property is multiprocessing safe.

Raises: RuntimeError – When the discipline counters are disabled.

parameters: list[str]: The parameters of the machine learning algorithm.

property status: str: The status of the discipline.

transformer: TransformerType: The transformation strategy for data groups.

class gemseo.mlearning.core.calibration.MLAlgoCalibration(algo, dataset, parameters, calibration_space, measure, measure_options=None, transformer=None, **algo_options)[source]

Calibration of a machine learning algorithm.

Parameters

algo (str) – The name of a machine learning algorithm.
dataset (Dataset) – A learning dataset.
parameters (Iterable[str]) – The parameters of the machine learning algorithm to calibrate.
calibration_space (DesignSpace) – The space defining the calibration variables.
measure (MLQualityMeasure) – A measure to assess the machine learning algorithm.
measure_options (MeasureOptionsType | None) –
The options of the quality measure. If None, do not use the quality measure options.

By default it is set to None.
transformer (TransformerType | None) –
The transformation strategy for the data groups. If None, do not transform data.

By default it is set to None.
**algo_options (MLAlgoParameterType) – The options of the machine learning algorithm.

Return type

None

execute(input_data)[source]

Calibrate the machine learning algorithm from a driver.

The driver can be either a DOE or an optimizer.

Parameters: input_data (Mapping[str, Union[str, int, Mapping[str, Union[int, float]]]]) – The driver properties.
Return type: None

get_history(name)[source]

Return the history of a variable.

Parameters: name (str) – The name of the variable.
Returns: The history of the variable.
Return type: numpy.ndarray

algo_assessor: gemseo.mlearning.core.calibration.MLAlgoAssessor: The assessor for the machine learning algorithm.

property algos: gemseo.mlearning.core.ml_algo.MLAlgo: The trained machine learning algorithms.

calibration_space: gemseo.algos.design_space.DesignSpace: The space defining the calibration variables.

dataset: gemseo.core.dataset.Dataset: The learning dataset.

maximize_objective: bool: Whether to maximize the quality measure.

optimal_algorithm: gemseo.mlearning.core.ml_algo.MLAlgo: The optimal machine learning algorithm.

optimal_criterion: float: The optimal quality measure.

optimal_parameters: dict[str, numpy.ndarray]: The optimal parameters for the machine learning algorithm.

scenario: gemseo.core.scenario.Scenario: The scenario used to calibrate the machine learning algorithm.

Examples

This module contains a class to select a machine learning algorithm from a list.

Machine learning is used to find relations or underlying structures in data. There is however no algorithm that is universally better than the others for an arbitrary problem. As for optimization, there is no free lunch for machine learning [Wol96].

Provided a quality measure, one can thus compare the performances of different machine learning algorithms.

This process can be easily performed using the class MLAlgoSelection.

A machine learning algorithm is built using a set of (hyper)parameters, before the learning takes place. In order to choose the best hyperparameters, a simple grid search over different values may be sufficient. The MLAlgoSelection does this. It can also perform a more advanced form of optimization than a simple grid search over predefined values, using the class MLAlgoCalibration.

See also

ml_algo calibration

class gemseo.mlearning.core.selection.MLAlgoSelection(dataset, measure, eval_method='learn', samples=None, **measure_options)[source]

Machine learning algorithm selector.

Parameters

dataset (Dataset) – The learning dataset.
measure (str | MLQualityMeasure) – The name of a quality measure to measure the quality of the machine learning algorithms.
eval_method (str) –
The name of the method to evaluate the quality measure.

By default it is set to learn.
samples (Sequence[int] | None) –
The indices of the learning samples to consider. Other indices are neither used for training nor for testing. If None, use all the samples.

By default it is set to None.
**measure_options (MeasureOptionType) – The options for the method to evaluate the quality measure. The option ‘multioutput’ will be set to False.

Raises

ValueError – If the unsupported “multioutput” option is enabled.

Return type

None

add_candidate(name, calib_space=None, calib_algo=None, **option_lists)[source]

Add a machine learning algorithm candidate.

Parameters

name (str) – The name of a machine learning algorithm.
calib_space (DesignSpace | None) –
The design space defining the parameters to be calibrated with a MLAlgoCalibration. If None, do not perform calibration.

By default it is set to None.
calib_algo (ScenarioInputDataType | None) –
The name and the parameters of the optimization algorithm, e.g. {“algo”: “fullfact”, “n_samples”: 10}. If None, do not perform calibration.

By default it is set to None.
**option_lists – The parameters for the machine learning algorithm candidate. Each parameter has to be enclosed within a list. The list may contain different values to try out for the given parameter, or only one.

Return type

None

Examples

>>> selector.add_candidate(
>>>     "LinearRegressor",
>>>     penalty_level=[0, 0.1, 1, 10, 20],
>>>     l2_penalty_ratio=[0, 0.5, 1],
>>>     fit_intercept=[True],
>>> )

select(return_quality=False)[source]

Select the best model.

The model is chosen through a grid search over candidates and their options, as well as an eventual optimization over the parameters in the calibration space.

Parameters

return_quality (bool) –

Whether to return the quality of the best model.

By default it is set to False.

Returns

The best model and its quality if required.

Return type

MLAlgo | tuple[MLAlgo, float]

candidates: list[tuple[MLAlgo, float]]: The candidate machine learning algorithms, after possible calibration, and their quality measures.

dataset: Dataset: The learning dataset.

factory: MLAlgoFactory: The factory used for the instantiation of machine learning algorithms.

measure: str: The name of a quality measure to measure the quality of the machine learning algorithms.

measure_options: dict[str, int | Dataset]: The options for the method to evaluate the quality measure.

Examples