:orphan:

..
   Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

   This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
   International License. To view a copy of this license, visit
   http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
   Commons, PO Box 1866, Mountain View, CA 94042, USA.

.. _machine_learning_examples:

Machine learning
================

This section illustrates the features of the :mod:`gemseo.mlearning` package:

- how to create a machine learning model for classification, clustering or regression,
- how to transform the data,
- how to assess the quality of a model,
- how to tune a model.

This package is particularly useful for creating a surrogate model,
as a :class:`.SurrogateDiscipline` wraps a regression model built from it.
However,
for those who are not very comfortable with machine learning
and for those who are essentially interested in the surrogate modelling,
starting with :ref:`the examples <surrogate_discipline_examples>`
about the :class:`.SurrogateDiscipline` should be more relevant.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. thumbnail-parent-div-close

.. raw:: html

    </div>

..
   Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

   This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
   International License. To view a copy of this license, visit
   http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
   Commons, PO Box 1866, Mountain View, CA 94042, USA.

Calibration and selection
~~~~~~~~~~~~~~~~~~~~~~~~~

During the training stage,
the parameters of a machine learning model are modified
so that this model learns the training data as well as possible.

This model also depends on hyperparameters that are fixed during training.
For example, the polynomial degree in the case of polynomial regression.
The :class:`.MLAlgoCalibration` class can be used
to tune these hyperparameters so as to improve this model.

Moreover,
even if this model has learned well,
it is possible that another has learned better.
The :class:`.MLAlgoSelection` class can be used
to select the best machine learning model from a collection.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Calibration of a polynomial regression">

.. only:: html

  .. image:: /examples/mlearning/calibration/images/thumb/sphx_glr_plot_calibration_thumb.png
    :alt:

  :doc:`/examples/mlearning/calibration/plot_calibration`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Calibration of a polynomial regression</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="In this example we use the .MLAlgoSelection class to perform a grid search over different algorithms and hyperparameter values.">

.. only:: html

  .. image:: /examples/mlearning/calibration/images/thumb/sphx_glr_plot_selection_thumb.png
    :alt:

  :doc:`/examples/mlearning/calibration/plot_selection`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Machine learning algorithm selection example</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /examples/mlearning/calibration/plot_calibration
   /examples/mlearning/calibration/plot_selection

..
   Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

   This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
   International License. To view a copy of this license, visit
   http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
   Commons, PO Box 1866, Mountain View, CA 94042, USA.

Classification
~~~~~~~~~~~~~~

The purpose of a classification model is
to predict the label value corresponding to a feature value
after being trained from several feature-label samples.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="The gemseo.mlearning package includes high-level functions to create classification models from model class names.">

.. only:: html

  .. image:: /examples/mlearning/classification_model/images/thumb/sphx_glr_plot_classification_api_thumb.png
    :alt:

  :doc:`/examples/mlearning/classification_model/plot_classification_api`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">High-level functions</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .RandomForestClassifier is a random forest model based on scikit-learn.">

.. only:: html

  .. image:: /examples/mlearning/classification_model/images/thumb/sphx_glr_plot_random_forest_classification_thumb.png
    :alt:

  :doc:`/examples/mlearning/classification_model/plot_random_forest_classification`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Random forest</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .KNNClassifier is a k-nearest neighbors model based on scikit-learn.">

.. only:: html

  .. image:: /examples/mlearning/classification_model/images/thumb/sphx_glr_plot_knn_thumb.png
    :alt:

  :doc:`/examples/mlearning/classification_model/plot_knn`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">k-nearest neighbors</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /examples/mlearning/classification_model/plot_classification_api
   /examples/mlearning/classification_model/plot_random_forest_classification
   /examples/mlearning/classification_model/plot_knn

..
   Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

   This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
   International License. To view a copy of this license, visit
   http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
   Commons, PO Box 1866, Mountain View, CA 94042, USA.

Clustering
~~~~~~~~~~

The purpose of a clustering model is
to group training observations according to their similarities.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .GaussianMixture is a Gaussian mixture model based on scikit-learn.">

.. only:: html

  .. image:: /examples/mlearning/clustering_model/images/thumb/sphx_glr_plot_gaussian_mixture_thumb.png
    :alt:

  :doc:`/examples/mlearning/clustering_model/plot_gaussian_mixture`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Gaussian mixture</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="The gemseo.mlearning package includes high-level functions to create clustering models from model class names.">

.. only:: html

  .. image:: /examples/mlearning/clustering_model/images/thumb/sphx_glr_plot_clustering_api_thumb.png
    :alt:

  :doc:`/examples/mlearning/clustering_model/plot_clustering_api`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">High-level functions</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .KMeans is a k-means model based on scikit-learn.">

.. only:: html

  .. image:: /examples/mlearning/clustering_model/images/thumb/sphx_glr_plot_kmeans_thumb.png
    :alt:

  :doc:`/examples/mlearning/clustering_model/plot_kmeans`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">K-means</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /examples/mlearning/clustering_model/plot_gaussian_mixture
   /examples/mlearning/clustering_model/plot_clustering_api
   /examples/mlearning/clustering_model/plot_kmeans

..
   Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

   This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
   International License. To view a copy of this license, visit
   http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
   Commons, PO Box 1866, Mountain View, CA 94042, USA.

Dimension reduction
~~~~~~~~~~~~~~~~~~~

How to reduce the dimension of a high-dimensional variable.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="In this demo, we apply a mixture of experts regression model to the Burgers dataset. In order to reduce the output dimension, we apply a PCA to the outputs.">

.. only:: html

  .. image:: /examples/mlearning/dimension_reduction/images/thumb/sphx_glr_plot_moe_burgers_thumb.png
    :alt:

  :doc:`/examples/mlearning/dimension_reduction/plot_moe_burgers`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Mixture of experts with PCA</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Example using PCA on solutions of the Burgers equation.">

.. only:: html

  .. image:: /examples/mlearning/dimension_reduction/images/thumb/sphx_glr_plot_pca_burgers_thumb.png
    :alt:

  :doc:`/examples/mlearning/dimension_reduction/plot_pca_burgers`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Principal component analysis (PCA)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Example using KL-SVD on solutions of the Burgers equation.">

.. only:: html

  .. image:: /examples/mlearning/dimension_reduction/images/thumb/sphx_glr_plot_klsvd_burgers_thumb.png
    :alt:

  :doc:`/examples/mlearning/dimension_reduction/plot_klsvd_burgers`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Proper orthogonal decomposition (POD)</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /examples/mlearning/dimension_reduction/plot_moe_burgers
   /examples/mlearning/dimension_reduction/plot_pca_burgers
   /examples/mlearning/dimension_reduction/plot_klsvd_burgers

..
   Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

   This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
   International License. To view a copy of this license, visit
   http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
   Commons, PO Box 1866, Mountain View, CA 94042, USA.

Quality
~~~~~~~

It is important to
evaluate the quality of a machine learning model
before using it.
|g| proposes numerical measures and visualizations for this purpose.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Cross-validation">

.. only:: html

  .. image:: /examples/mlearning/quality_measure/images/thumb/sphx_glr_plot_cross_validation_thumb.png
    :alt:

  :doc:`/examples/mlearning/quality_measure/plot_cross_validation`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Cross-validation</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Error from surrogate discipline">

.. only:: html

  .. image:: /examples/mlearning/quality_measure/images/thumb/sphx_glr_plot_from_surrogate_thumb.png
    :alt:

  :doc:`/examples/mlearning/quality_measure/plot_from_surrogate`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Error from surrogate discipline</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Leave-one-out">

.. only:: html

  .. image:: /examples/mlearning/quality_measure/images/thumb/sphx_glr_plot_leave_one_out_thumb.png
    :alt:

  :doc:`/examples/mlearning/quality_measure/plot_leave_one_out`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Leave-one-out</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="MSE for regression models">

.. only:: html

  .. image:: /examples/mlearning/quality_measure/images/thumb/sphx_glr_plot_mse_thumb.png
    :alt:

  :doc:`/examples/mlearning/quality_measure/plot_mse`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">MSE for regression models</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="R2 for regression models">

.. only:: html

  .. image:: /examples/mlearning/quality_measure/images/thumb/sphx_glr_plot_r2_thumb.png
    :alt:

  :doc:`/examples/mlearning/quality_measure/plot_r2`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">R2 for regression models</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="RMSE for regression models">

.. only:: html

  .. image:: /examples/mlearning/quality_measure/images/thumb/sphx_glr_plot_rmse_thumb.png
    :alt:

  :doc:`/examples/mlearning/quality_measure/plot_rmse`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">RMSE for regression models</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /examples/mlearning/quality_measure/plot_cross_validation
   /examples/mlearning/quality_measure/plot_from_surrogate
   /examples/mlearning/quality_measure/plot_leave_one_out
   /examples/mlearning/quality_measure/plot_mse
   /examples/mlearning/quality_measure/plot_r2
   /examples/mlearning/quality_measure/plot_rmse

..
   Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

   This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
   International License. To view a copy of this license, visit
   http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
   Commons, PO Box 1866, Mountain View, CA 94042, USA.

Regression
~~~~~~~~~~

The purpose of a regression model is
to predict the output value corresponding to an input value
after being trained from several input-output samples.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Advanced mixture of experts">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_advanced_moe_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_advanced_moe`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Advanced mixture of experts</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Given a training dataset whose input samples are generated from OpenTURNS probability distributions, the .FCERegressor can use any linear model fitting algorithm, including sparse techniques, to fit a functional chaos expansion (FCE) model of the form">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_fce_regression_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_fce_regression`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Function chaos expansion</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .GaussianProcessRegressor is a GP regression model based on scikit-learn.">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_gp_regression_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_gp_regression`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Gaussian process (GP) regression</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="The gemseo.mlearning package includes high-level functions to create regression models from model class names.">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_regression_api_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_regression_api`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">High-level functions</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .LinearRegressor is a linear regression model based on scikit-learn.">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_linear_regression_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_linear_regression`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Linear regression</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .MOERegressor is a mixture of experts for regression purposes.">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_moe_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_moe`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Mixture of experts</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .PCERegressor is a PCE model based on OpenTURNS.">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_pce_regression_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_pce_regression`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Polynomial chaos expansion (PCE)</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .PolynomialRegressor is a polynomial regression model based on a .LinearRegressor. This design choice was made because a polynomial regression model is a generalized linear model whose basis functions are monomials. Thus, a .PolynomialRegressor benefits from the same settings as .LinearRegressor: offset can be set to zero and regularization techniques can be used.">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_polynomial_regression_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_polynomial_regression`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Polynomial regression</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="An .RBFRegressor is an RBF model based on SciPy.">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_rbf_regression_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_rbf_regression`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Radial basis function (RBF) regression</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A .RandomForestRegressor is a random forest model based on scikit-learn.">

.. only:: html

  .. image:: /examples/mlearning/regression_model/images/thumb/sphx_glr_plot_random_forest_regression_thumb.png
    :alt:

  :doc:`/examples/mlearning/regression_model/plot_random_forest_regression`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Random forest</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /examples/mlearning/regression_model/plot_advanced_moe
   /examples/mlearning/regression_model/plot_fce_regression
   /examples/mlearning/regression_model/plot_gp_regression
   /examples/mlearning/regression_model/plot_regression_api
   /examples/mlearning/regression_model/plot_linear_regression
   /examples/mlearning/regression_model/plot_moe
   /examples/mlearning/regression_model/plot_pce_regression
   /examples/mlearning/regression_model/plot_polynomial_regression
   /examples/mlearning/regression_model/plot_rbf_regression
   /examples/mlearning/regression_model/plot_random_forest_regression

..
   Copyright 2021 IRT Saint Exupéry, https://www.irt-saintexupery.com

   This work is licensed under the Creative Commons Attribution-ShareAlike 4.0
   International License. To view a copy of this license, visit
   http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative
   Commons, PO Box 1866, Mountain View, CA 94042, USA.

Data transformation
~~~~~~~~~~~~~~~~~~~

Fitting a model from transformed data rather than raw data can facilitate the training
and improve the quality of the machine learning model.
Every machine learning model has a ``transformer`` argument to set the transformation policy (none by default).
In the special case of regression models,
the function :func:`.create_surrogate` and the :class:`.SurrogateDiscipline` class
use the :attr:`.BaseRegressor.DEFAULT_TRANSFORMER` by default,
which is :class:`.MinMaxScaler` for both inputs and outputs.


.. raw:: html

  <div id='sg-tag-list' class='sphx-glr-tag-list'></div>


.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Pipeline">

.. only:: html

  .. image:: /examples/mlearning/transformer/images/thumb/sphx_glr_plot_pipeline_thumb.png
    :alt:

  :doc:`/examples/mlearning/transformer/plot_pipeline`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Pipeline</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Scalers">

.. only:: html

  .. image:: /examples/mlearning/transformer/images/thumb/sphx_glr_plot_scaler_thumb.png
    :alt:

  :doc:`/examples/mlearning/transformer/plot_scaler`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Scalers</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Scaling">

.. only:: html

  .. image:: /examples/mlearning/transformer/images/thumb/sphx_glr_plot_scaling_data_thumb.png
    :alt:

  :doc:`/examples/mlearning/transformer/plot_scaling_data`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Scaling</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /examples/mlearning/transformer/plot_pipeline
   /examples/mlearning/transformer/plot_scaler
   /examples/mlearning/transformer/plot_scaling_data


.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-gallery

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download all examples in Python source code: mlearning_python.zip </examples/mlearning/mlearning_python.zip>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download all examples in Jupyter notebooks: mlearning_jupyter.zip </examples/mlearning/mlearning_jupyter.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_