Calibrate or select a machine learning algorithm¶

Calibration of a machine learning algorithm.

A machine learning algorithm depends on hyper-parameters, e.g. the number of clusters for a clustering algorithm, the regularization constant for a regression model, the kernel for a Gaussian process regression, … Its ability to generalize the information learned during the training stage, and thus to avoid over-fitting, which is an over-reliance on the learning data set, depends on the values of these hyper-parameters. Thus, the hyper-parameters minimizing the learning quality measure are rarely those minimizing the generalization one. Classically, the generalization one decreases before growing again as the model becomes more complex, while the learning error keeps decreasing. This phenomenon is called the curse of dimensionality.

In this module, the MLAlgoCalibration class aims to calibrate the hyper-parameters in order to minimize this measure of the generalization quality over a calibration parameter space. This class relies on the MLAlgoAssessor class which is a discipline (MDODiscipline) built from a machine learning algorithm (MLAlgo), a dataset (Dataset), a quality measure (MLQualityMeasure) and various options for the data scaling, the quality measure and the machine learning algorithm. The inputs of this discipline are hyper-parameters of the machine learning algorithm while the output is the quality criterion.

Classes:

`MLAlgoAssessor`(algo, dataset, parameters, …)	Discipline assessing the quality of a machine learning algorithm.
`MLAlgoCalibration`(algo, dataset, parameters, …)	Calibration of a machine learning algorithm.

class gemseo.mlearning.core.calibration.MLAlgoAssessor(algo, dataset, parameters, measure, measure_options=None, transformer=None, **algo_options)[source]

Discipline assessing the quality of a machine learning algorithm.

This quality depends on the values of parameters to calibrate with the MLAlgoCalibration.

Attributes

algo (str) – The name of a machine learning algorithm.
measure (MLQualityMeasure) – The measure to assess the machine learning algorithm.
measure_options (Dict[str,Union[int,Dataset]]) – The options of the quality measure.
parameters (List(str)) – The parameters of the machine learning algorithm to calibrate.
dataset (Dataset) – The learning dataset.
transformer (TransformerType) – The transformation strategy for data groups.
algos (List(MLAlgo)) – The instances of the machine learning algorithm (one per execution of the machine learning algorithm assessor).

Parameters

algo (str) – The name of a machine learning algorithm.
dataset (Dataset) – A learning dataset.
parameters (Iterable[str]) – The parameters of the machine learning algorithm to calibrate.
measure (MLQualityMeasure) – A measure to assess the machine learning algorithm.
measure_options (Optional[MeasureOptionsType]) – The options of the quality measure. If “multioutput” is missing, it is added with False as value. If None, do not use quality measure options.
transformer (Dict[str,Transformer]) – The strategies to transform the variables. The values are instances of Transformer while the keys are the names of either the variables or the groups of variables, e.g. “inputs” or “outputs” in the case of the regression algorithms. If a group is specified, the Transformer will be applied to all the variables of this group. If None, do not transform the variables.
**options – The options of the machine learning algorithm.
algo_options (MLAlgoParameterType) –

Raises

ValueError – If the measure option “multioutput” is True.

Return type

None

Methods:

`activate_time_stamps`()	Activate the time stamps.
`add_differentiated_inputs`([inputs])	Add inputs to the differentiation list.
`add_differentiated_outputs`([outputs])	Add outputs to the differentiation list.
`add_status_observer`(obs)	Add an observer for the status.
`auto_get_grammar_file`([is_input, name, comp_dir])	Use a naming convention to associate a grammar file to a discipline.
`check_input_data`(input_data[, raise_exception])	Check the input data validity.
`check_jacobian`([input_data, derr_approx, …])	Check if the jacobian provided by the linearize() method is correct.
`check_output_data`([raise_exception])	Check the output data validity.
`deactivate_time_stamps`()	Deactivate the time stamps for storing start and end times of execution and linearizations.
`deserialize`(in_file)	Derialize the discipline from a file.
`execute`([input_data])	Execute the discipline.
`get_all_inputs`()	Accessor for the input data as a list of values.
`get_all_outputs`()	Accessor for the output data as a list of values.
`get_attributes_to_serialize`()	Define the attributes to be serialized.
`get_data_list_from_dict`(keys, data_dict)	Filter the dict from a list of keys or a single key.
`get_expected_dataflow`()	Return the expected data exchange sequence.
`get_expected_workflow`()	Return the expected execution sequence.
`get_input_data`()	Accessor for the input data as a dict of values.
`get_input_data_names`()	Accessor for the input names as a list.
`get_input_output_data_names`()	Accessor for the input and output names as a list.
`get_inputs_asarray`()	Accessor for the outputs as a large numpy array.
`get_inputs_by_name`(data_names)	Accessor for the inputs as a list.
`get_local_data_by_name`(data_names)	Accessor for the local data of the discipline as a dict of values.
`get_output_data`()	Accessor for the output data as a dict of values.
`get_output_data_names`()	Accessor for the output names as a list.
`get_outputs_asarray`()	Accessor for the outputs as a large numpy array.
`get_outputs_by_name`(data_names)	Accessor for the outputs as a list.
`get_sub_disciplines`()	Gets the sub disciplines of self By default, empty.
`is_all_inputs_existing`(data_names)	Test if all the names in data_names are inputs of the discipline.
`is_all_outputs_existing`(data_names)	Test if all the names in data_names are outputs of the discipline.
`is_input_existing`(data_name)	Test if input named data_name is an input of the discipline.
`is_output_existing`(data_name)	Test if output named data_name is an output of the discipline.
`is_scenario`()	Return True if self is a scenario.
`linearize`([input_data, force_all, force_no_exec])	Execute the linearized version of the code.
`notify_status_observers`()	Notify all status observers that the status has changed.
`remove_status_observer`(obs)	Remove an observer for the status.
`reset_statuses_for_run`()	Sets all the statuses to PENDING.
`serialize`(out_file)	Serialize the discipline.
`set_cache_policy`([cache_type, …])	Set the type of cache to use and the tolerance level.
`set_disciplines_statuses`(status)	Set the sub disciplines statuses.
`set_jacobian_approximation`([…])	Set the jacobian approximation method.
`set_optimal_fd_step`([outputs, inputs, …])	Compute the optimal finite-difference step.
`store_local_data`(**kwargs)	Store discipline data in local data.

Attributes:

`cache_tol`	Accessor to the cache input tolerance.
`default_inputs`	Accessor to the default inputs.
`exec_time`	Return the cumulated execution time.
`linearization_mode`	Accessor to the linearization mode.
`n_calls`	Return the number of calls to execute() which triggered the _run().
`n_calls_linearize`	Return the number of calls to linearize() which triggered the _compute_jacobian() method.
`status`	Status accessor.

classmethod activate_time_stamps()

Activate the time stamps.

For storing start and end times of execution and linearizations.

add_differentiated_inputs(inputs=None)

Add inputs to the differentiation list.

This method updates self._differentiated_inputs with inputs

Parameters: inputs – list of inputs variables to differentiate if None, all inputs of discipline are used (Default value = None)

add_differentiated_outputs(outputs=None)

Add outputs to the differentiation list.

Update self._differentiated_inputs with inputs.

Parameters: outputs – list of output variables to differentiate if None, all outputs of discipline are used

add_status_observer(obs)

Add an observer for the status.

Add an observer for the status to be notified when self changes of status.

Parameters: obs – the observer to add

auto_get_grammar_file(is_input=True, name=None, comp_dir=None)

Use a naming convention to associate a grammar file to a discipline.

This method searches in the “comp_dir” directory containing the discipline source file for files basenames self.name _input.json and self.name _output.json

Parameters

is_input – if True, searches for _input.json, otherwise _output.json (Default value = True)
name – the name of the discipline (Default value = None)
comp_dir – the containing directory if None, use self.comp_dir (Default value = None)

Returns

path to the grammar file

Return type

string

property cache_tol: Accessor to the cache input tolerance.

check_input_data(input_data, raise_exception=True)

Check the input data validity.

Parameters

input_data – the input data dict
raise_exception – Default value = True)

check_jacobian(input_data=None, derr_approx='finite_differences', step=1e-07, threshold=1e-08, linearization_mode='auto', inputs=None, outputs=None, parallel=False, n_processes=2, use_threading=False, wait_time_between_fork=0, auto_set_step=False, plot_result=False, file_path='jacobian_errors.pdf', show=False, figsize_x=10, figsize_y=10)

Check if the jacobian provided by the linearize() method is correct.

Parameters

input_data – input data dict (Default value = None)
derr_approx – derivative approximation method: COMPLEX_STEP (Default value = COMPLEX_STEP)
threshold – acceptance threshold for the jacobian error (Default value = 1e-8)
linearization_mode – the mode of linearization: direct, adjoint or automated switch depending on dimensions of inputs and outputs (Default value = ‘auto’)
inputs – list of inputs wrt which to differentiate (Default value = None)
outputs – list of outputs to differentiate (Default value = None)
step – the step for finite differences or complex step
parallel – if True, executes in parallel
n_processes – maximum number of processors on which to run
use_threading – if True, use Threads instead of processes to parallelize the execution multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory This is important to note if you want to execute the same discipline multiple times, you shall use multiprocessing
wait_time_between_fork – time waited between two forks of the process /Thread
auto_set_step – Compute optimal step for a forward first order finite differences gradient approximation
plot_result – plot the result of the validation (computed and approximate jacobians)
file_path – path to the output file if plot_result is True
show – if True, open the figure
figsize_x – x size of the figure in inches
figsize_y – y size of the figure in inches

Returns

True if the check is accepted, False otherwise

check_output_data(raise_exception=True)

Check the output data validity.

Parameters: raise_exception – if true, an exception is raised when data is invalid (Default value = True)

classmethod deactivate_time_stamps(): Deactivate the time stamps for storing start and end times of execution and linearizations.

property default_inputs: Accessor to the default inputs.

static deserialize(in_file)

Derialize the discipline from a file.

Parameters: in_file – input file for serialization
Returns: a discipline instance

property exec_time

Return the cumulated execution time.

Multiprocessing safe.

execute(input_data=None)

Execute the discipline.

This method executes the discipline:

Adds default inputs to the input_data if some inputs are not defined
in input_data but exist in self._default_data
Checks if the last execution of the discipline wan not called with
identical inputs, cached in self.cache, if yes, directly return self.cache.get_output_cache(inputs)
Caches the inputs
Checks the input data against self.input_grammar
if self.data_processor is not None: runs the preprocessor
updates the status to RUNNING
calls the _run() method, that shall be defined
if self.data_processor is not None: runs the postprocessor
checks the output data
Caches the outputs
updates the status to DONE or FAILED
updates summed execution time

Parameters: input_data (dict) – the input data dict needed to execute the disciplines according to the discipline input grammar (Default value = None)
Returns: the discipline local data after execution
Return type: dict

get_all_inputs()

Accessor for the input data as a list of values.

The order is given by self.get_input_data_names().

Returns: the data

get_all_outputs()

Accessor for the output data as a list of values.

The order is given by self.get_output_data_names().

Returns: the data

get_attributes_to_serialize()

Define the attributes to be serialized.

Shall be overloaded by disciplines

Returns: the list of attributes names
Return type: list

static get_data_list_from_dict(keys, data_dict)

Filter the dict from a list of keys or a single key.

If keys is a string, then the method return the value associated to the key. If keys is a list of string, then the method return a generator of value corresponding to the keys which can be iterated.

Parameters

keys – a sting key or a list of keys
data_dict – the dict to get the data from

Returns

a data or a generator of data

get_expected_dataflow()

Return the expected data exchange sequence.

This method is used for the XDSM representation.

Default to empty list See MDOFormulation.get_expected_dataflow

Returns: a list representing the data exchange arcs

get_expected_workflow()

Return the expected execution sequence.

This method is used for XDSM representation Default to the execution of the discipline itself See MDOFormulation.get_expected_workflow

get_input_data()

Accessor for the input data as a dict of values.

Returns: the data dict

get_input_data_names()

Accessor for the input names as a list.

Returns: the data names list

get_input_output_data_names()

Accessor for the input and output names as a list.

Returns: the data names list

get_inputs_asarray()

Accessor for the outputs as a large numpy array.

The order is the one of self.get_all_outputs().

Returns: the outputs array
Return type: ndarray

get_inputs_by_name(data_names)

Accessor for the inputs as a list.

Parameters: data_names – the data names list
Returns: the data list

get_local_data_by_name(data_names)

Accessor for the local data of the discipline as a dict of values.

Parameters: data_names – the names of the data which will be the keys of the dictionary
Returns: the data list

get_output_data()

Accessor for the output data as a dict of values.

Returns: the data dict

get_output_data_names()

Accessor for the output names as a list.

Returns: the data names list

get_outputs_asarray()

Accessor for the outputs as a large numpy array.

The order is the one of self.get_all_outputs()

Returns: the outputs array
Return type: ndarray

get_outputs_by_name(data_names)

Accessor for the outputs as a list.

Parameters: data_names – the data names list
Returns: the data list

get_sub_disciplines()

Gets the sub disciplines of self By default, empty.

Returns: the list of disciplines

is_all_inputs_existing(data_names)

Test if all the names in data_names are inputs of the discipline.

Parameters: data_names – the names of the inputs
Returns: True if data_names are all in input grammar
Return type: logical

is_all_outputs_existing(data_names)

Test if all the names in data_names are outputs of the discipline.

Parameters: data_names – the names of the outputs
Returns: True if data_names are all in output grammar
Return type: logical

is_input_existing(data_name)

Test if input named data_name is an input of the discipline.

Parameters: data_name – the name of the output
Returns: True if data_name is in input grammar
Return type: logical

is_output_existing(data_name)

Test if output named data_name is an output of the discipline.

Parameters: data_name – the name of the output
Returns: True if data_name is in output grammar
Return type: logical

static is_scenario()

Return True if self is a scenario.

Returns: True if self is a scenario

property linearization_mode: Accessor to the linearization mode.

linearize(input_data=None, force_all=False, force_no_exec=False)

Execute the linearized version of the code.

Parameters

input_data – the input data dict needed to execute the disciplines according to the discipline input grammar
force_all – if False, self._differentiated_inputs and self.differentiated_output are used to filter the differentiated variables otherwise, all outputs are differentiated wrt all inputs (Default value = False)
force_no_exec – if True, the discipline is not re executed, cache is loaded anyway

property n_calls

Return the number of calls to execute() which triggered the _run().

Multiprocessing safe.

property n_calls_linearize

Return the number of calls to linearize() which triggered the _compute_jacobian() method.

Multiprocessing safe.

notify_status_observers(): Notify all status observers that the status has changed.

remove_status_observer(obs)

Remove an observer for the status.

Parameters: obs – the observer to remove

reset_statuses_for_run(): Sets all the statuses to PENDING.

serialize(out_file)

Serialize the discipline.

Parameters: out_file – destination file for serialization

set_cache_policy(cache_type='SimpleCache', cache_tolerance=0.0, cache_hdf_file=None, cache_hdf_node_name=None, is_memory_shared=True)

Set the type of cache to use and the tolerance level.

This method set the cache policy to cache data whose inputs are close to inputs whose outputs are already cached. The cache can be either a simple cache recording the last execution or a full cache storing all executions. Caching data can be either in-memory, e.g. SimpleCache and MemoryFullCache , or on the disk, e.g. HDF5Cache . CacheFactory.caches provides the list of available types of caches.

Parameters

cache_type (str) – type of cache to use.
cache_tolerance (float) – tolerance for the approximate cache maximal relative norm difference to consider that two input arrays are equal
cache_hdf_file (str) – the file to store the data, mandatory when HDF caching is used
cache_hdf_node_name (str) – name of the HDF dataset to store the discipline data. If None, self.name is used
is_memory_shared (bool) – If True, a shared memory dict is used to store the data, which makes the cache compatible with multiprocessing. WARNING: if set to False, and multiple disciplines point to the same cache or the process is multiprocessed, there may be duplicate computations because the cache will not be shared among the processes.

set_disciplines_statuses(status)

Set the sub disciplines statuses.

To be implemented in subclasses. :param status: the status

set_jacobian_approximation(jac_approx_type='finite_differences', jax_approx_step=1e-07, jac_approx_n_processes=1, jac_approx_use_threading=False, jac_approx_wait_time=0)

Set the jacobian approximation method.

Sets the linearization mode to approx_method, sets the parameters of the approximation for further use when calling self.linearize

Parameters

jac_approx_type – “complex_step” or “finite_differences”
jax_approx_step – the step for finite differences or complex step
jac_approx_n_processes – maximum number of processors on which to run
jac_approx_use_threading – if True, use Threads instead of processes to parallelize the execution multiprocessing will copy (serialize) all the disciplines, while threading will share all the memory This is important to note if you want to execute the same discipline multiple times, you shall use multiprocessing
jac_approx_wait_time – time waited between two forks of the process /Thread

set_optimal_fd_step(outputs=None, inputs=None, force_all=False, print_errors=False, numerical_error=2.220446049250313e-16)

Compute the optimal finite-difference step.

Compute the optimal step for a forward first order finite differences gradient approximation. Requires a first evaluation of perturbed functions values. The optimal step is reached when the truncation error (cut in the Taylor development), and the numerical cancellation errors (roundoff when doing f(x+step)-f(x)) are approximately equal.

Warning: this calls the discipline execution two times per input variables.

See: https://en.wikipedia.org/wiki/Numerical_differentiation and “Numerical Algorithms and Digital Representation”, Knut Morken , Chapter 11, “Numerical Differenciation”

Parameters

inputs – inputs wrt the linearization is made. If None, use differentiated inputs
outputs – outputs of the linearization is made. If None, use differentiated outputs
force_all – if True, all inputs and outputs are used
print_errors – if True, displays the estimated errors
numerical_error – numerical error associated to the calculation of f. By default Machine epsilon (appx 1e-16), but can be higher when the calculation of f requires a numerical resolution

Returns

the estimated errors of truncation and cancelation error.

property status: Status accessor.

store_local_data(**kwargs)

Store discipline data in local data.

Parameters: kwargs – the data as key value pairs

class gemseo.mlearning.core.calibration.MLAlgoCalibration(algo, dataset, parameters, calibration_space, measure, measure_options=None, transformer=None, **algo_options)[source]

Calibration of a machine learning algorithm.

Attributes

algo_assessor (MLAlgoAssessor) – The assessor for the machine learning algorithm.
calibration_space (DesignSpace) – The space defining the calibration variables.
maximize_objective (bool) – If True, seek to maximize the quality measure.
dataset (Dataset) – The learning dataset.
optimal_parameters (Dict[str,ndarray]) – The optimal parameters for the machine learning algorithm.
optimal_criterion (float) – The optimal quality measure.
optimal_algorithm (MLAlgo) – The optimal machine learning algorithm.
scenario (Scenario) – The scenario used to calibrate the machine learning algorithm.

Parameters

algo (str) – The name of a machine learning algorithm.
dataset (Dataset) – A learning dataset.
parameters (Iterable[str]) – The parameters of the machine learning algorithm to calibrate.
calibration_space (DesignSpace) – The space defining the calibration variables.
measure (MLQualityMeasure) – A measure to assess the machine learning algorithm.
measure_options (Optional[MeasureOptionsType]) – The options of the quality measure. If None, do not use the quality measure options.
transformer (Optional[TransformerType]) – The transformation strategy for the data groups. If None, do not transform data.
**algo_options – The options of the machine learning algorithm.
algo_options (MLAlgoParameterType) –

Return type

None

Attributes:

algos

The trained machine learning algorithms.

Methods:

`execute`(input_data)	Calibrate the machine learning algorithm from a driver.
`get_history`(name)	Return the history of a variable.

property algos: The trained machine learning algorithms.

execute(input_data)[source]

Calibrate the machine learning algorithm from a driver.

The driver can be either a DOE or an optimizer.

Parameters: input_data (Mapping[str, Union[str, int, Mapping[str, Union[int, float]]]]) – The driver properties.
Return type: None

get_history(name)[source]

Return the history of a variable.

Parameters: name (str) – The name of the variable.
Returns: The history of the variable.
Return type: numpy.ndarray

Examples

This module contains a class to select a machine learning algorithm from a list.

Machine learning is used to find relations or underlying structures in data. There is however no algorithm that is universally better than the others for an arbitrary problem. As for optimization, there is no free lunch for machine learning [wolpert].

wolpert: Wolpert, David H. “The lack of a priori distinctions between learning algorithms.” Neural computation 8.7 (1996): 1341-1390.

Provided a quality measure, one can thus compare the performances of different machine learning algorithms.

This process can be easily performed using the class MLAlgoSelection.

A machine learning algorithm is built using a set of (hyper)parameters, before the learning takes place. In order to choose the best hyperparameters, a simple grid search over different values may be sufficient. The MLAlgoSelection does this. It can also perform a more advanced form of optimization than a simple grid search over predefined values, using the class MLAlgoCalibration.

`add_candidate`(name[, calib_space, calib_algo])	Add a machine learning algorithm candidate.
`select`([return_quality])	Select the best model.