.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "examples/mlearning/regression_model/plot_linear_regression.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_examples_mlearning_regression_model_plot_linear_regression.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_examples_mlearning_regression_model_plot_linear_regression.py:


Linear regression
=================

A :class:`.LinearRegressor` is a linear regression model
based on `scikit-learn <https://scikit-learn.org/>`__.

.. seealso::
   You can find more information about building linear models with scikit-learn on
   `this page <https://scikit-learn.org/stable/modules/linear_model.html>`__.

.. GENERATED FROM PYTHON SOURCE LINES 32-46

.. code-block:: Python


    from __future__ import annotations

    from matplotlib import pyplot as plt
    from numpy import array

    from gemseo import configure_logger
    from gemseo import create_design_space
    from gemseo import create_discipline
    from gemseo import sample_disciplines
    from gemseo.mlearning import create_regression_model

    configure_logger()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    <RootLogger root (INFO)>



.. GENERATED FROM PYTHON SOURCE LINES 47-52

Problem
-------
In this example,
we represent the function :math:`f(x)=(6x-2)^2\sin(12x-4)` :cite:`forrester2008`
by the :class:`.AnalyticDiscipline`

.. GENERATED FROM PYTHON SOURCE LINES 52-57

.. code-block:: Python

    discipline = create_discipline(
        "AnalyticDiscipline",
        name="f",
        expressions={"y": "(6*x-2)**2*sin(12*x-4)"},
    )







.. GENERATED FROM PYTHON SOURCE LINES 58-59

and seek to approximate it over the input space

.. GENERATED FROM PYTHON SOURCE LINES 59-62

.. code-block:: Python

    input_space = create_design_space()
    input_space.add_variable("x", lower_bound=0.0, upper_bound=1.0)








.. GENERATED FROM PYTHON SOURCE LINES 63-65

To do this,
we create a training dataset with 6 equispaced points:

.. GENERATED FROM PYTHON SOURCE LINES 65-69

.. code-block:: Python

    training_dataset = sample_disciplines(
        [discipline], input_space, "y", algo_name="PYDOE_FULLFACT", n_samples=6
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

     WARNING - 20:35:08: No coupling in MDA, switching chain_linearize to True.
        INFO - 20:35:08: *** Start Sampling execution ***
        INFO - 20:35:08: Sampling
        INFO - 20:35:08:    Disciplines: f
        INFO - 20:35:08:    MDO formulation: MDF
        INFO - 20:35:08: Running the algorithm PYDOE_FULLFACT:
        INFO - 20:35:08:     17%|█▋        | 1/6 [00:00<00:00, 660.52 it/sec]
        INFO - 20:35:08:     33%|███▎      | 2/6 [00:00<00:00, 1092.12 it/sec]
        INFO - 20:35:08:     50%|█████     | 3/6 [00:00<00:00, 1446.98 it/sec]
        INFO - 20:35:08:     67%|██████▋   | 4/6 [00:00<00:00, 1728.01 it/sec]
        INFO - 20:35:08:     83%|████████▎ | 5/6 [00:00<00:00, 1974.16 it/sec]
        INFO - 20:35:08:    100%|██████████| 6/6 [00:00<00:00, 2186.43 it/sec]
        INFO - 20:35:08: *** End Sampling execution ***




.. GENERATED FROM PYTHON SOURCE LINES 70-76

Basics
------
Training
~~~~~~~~
Then,
we train a linear regression model from these samples:

.. GENERATED FROM PYTHON SOURCE LINES 76-79

.. code-block:: Python

    model = create_regression_model("LinearRegressor", training_dataset)
    model.learn()








.. GENERATED FROM PYTHON SOURCE LINES 80-84

Prediction
~~~~~~~~~~
Once it is built,
we can predict the output value of :math:`f` at a new input point:

.. GENERATED FROM PYTHON SOURCE LINES 84-88

.. code-block:: Python

    input_value = {"x": array([0.65])}
    output_value = model.predict(input_value)
    output_value





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    {'y': array([3.29457456])}



.. GENERATED FROM PYTHON SOURCE LINES 89-90

as well as its Jacobian value:

.. GENERATED FROM PYTHON SOURCE LINES 90-93

.. code-block:: Python

    jacobian_value = model.predict_jacobian(input_value)
    jacobian_value





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    {'y': {'x': array([[7.26002643]])}}



.. GENERATED FROM PYTHON SOURCE LINES 94-98

Plotting
~~~~~~~~
Of course,
you can see that the linear model is no good at all here:

.. GENERATED FROM PYTHON SOURCE LINES 98-110

.. code-block:: Python

    test_dataset = sample_disciplines(
        [discipline], input_space, "y", algo_name="PYDOE_FULLFACT", n_samples=100
    )
    input_data = test_dataset.get_view(variable_names=model.input_names).to_numpy()
    reference_output_data = test_dataset.get_view(variable_names="y").to_numpy().ravel()
    predicted_output_data = model.predict(input_data).ravel()
    plt.plot(input_data.ravel(), reference_output_data, label="Reference")
    plt.plot(input_data.ravel(), predicted_output_data, label="Regression - Basics")
    plt.grid()
    plt.legend()
    plt.show()




.. image-sg:: /examples/mlearning/regression_model/images/sphx_glr_plot_linear_regression_001.png
   :alt: plot linear regression
   :srcset: /examples/mlearning/regression_model/images/sphx_glr_plot_linear_regression_001.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

     WARNING - 20:35:08: No coupling in MDA, switching chain_linearize to True.
        INFO - 20:35:08: *** Start Sampling execution ***
        INFO - 20:35:08: Sampling
        INFO - 20:35:08:    Disciplines: f
        INFO - 20:35:08:    MDO formulation: MDF
        INFO - 20:35:08: Running the algorithm PYDOE_FULLFACT:
        INFO - 20:35:08:      1%|          | 1/100 [00:00<00:00, 3986.98 it/sec]
        INFO - 20:35:08:      2%|▏         | 2/100 [00:00<00:00, 3907.13 it/sec]
        INFO - 20:35:08:      3%|▎         | 3/100 [00:00<00:00, 4076.10 it/sec]
        INFO - 20:35:08:      4%|▍         | 4/100 [00:00<00:00, 4128.25 it/sec]
        INFO - 20:35:08:      5%|▌         | 5/100 [00:00<00:00, 4217.92 it/sec]
        INFO - 20:35:08:      6%|▌         | 6/100 [00:00<00:00, 4285.73 it/sec]
        INFO - 20:35:08:      7%|▋         | 7/100 [00:00<00:00, 4348.36 it/sec]
        INFO - 20:35:08:      8%|▊         | 8/100 [00:00<00:00, 4406.94 it/sec]
        INFO - 20:35:08:      9%|▉         | 9/100 [00:00<00:00, 4407.84 it/sec]
        INFO - 20:35:08:     10%|█         | 10/100 [00:00<00:00, 4447.83 it/sec]
        INFO - 20:35:08:     11%|█         | 11/100 [00:00<00:00, 4489.38 it/sec]
        INFO - 20:35:08:     12%|█▏        | 12/100 [00:00<00:00, 4525.82 it/sec]
        INFO - 20:35:08:     13%|█▎        | 13/100 [00:00<00:00, 4558.64 it/sec]
        INFO - 20:35:08:     14%|█▍        | 14/100 [00:00<00:00, 4537.89 it/sec]
        INFO - 20:35:08:     15%|█▌        | 15/100 [00:00<00:00, 4561.67 it/sec]
        INFO - 20:35:08:     16%|█▌        | 16/100 [00:00<00:00, 4588.64 it/sec]
        INFO - 20:35:08:     17%|█▋        | 17/100 [00:00<00:00, 4616.88 it/sec]
        INFO - 20:35:08:     18%|█▊        | 18/100 [00:00<00:00, 4642.57 it/sec]
        INFO - 20:35:08:     19%|█▉        | 19/100 [00:00<00:00, 4631.63 it/sec]
        INFO - 20:35:08:     20%|██        | 20/100 [00:00<00:00, 4647.17 it/sec]
        INFO - 20:35:08:     21%|██        | 21/100 [00:00<00:00, 4659.35 it/sec]
        INFO - 20:35:08:     22%|██▏       | 22/100 [00:00<00:00, 4672.37 it/sec]
        INFO - 20:35:08:     23%|██▎       | 23/100 [00:00<00:00, 4674.11 it/sec]
        INFO - 20:35:08:     24%|██▍       | 24/100 [00:00<00:00, 4683.97 it/sec]
        INFO - 20:35:08:     25%|██▌       | 25/100 [00:00<00:00, 4699.82 it/sec]
        INFO - 20:35:08:     26%|██▌       | 26/100 [00:00<00:00, 4715.55 it/sec]
        INFO - 20:35:08:     27%|██▋       | 27/100 [00:00<00:00, 4728.05 it/sec]
        INFO - 20:35:08:     28%|██▊       | 28/100 [00:00<00:00, 4725.22 it/sec]
        INFO - 20:35:08:     29%|██▉       | 29/100 [00:00<00:00, 4731.95 it/sec]
        INFO - 20:35:08:     30%|███       | 30/100 [00:00<00:00, 4742.90 it/sec]
        INFO - 20:35:08:     31%|███       | 31/100 [00:00<00:00, 4752.67 it/sec]
        INFO - 20:35:08:     32%|███▏      | 32/100 [00:00<00:00, 4765.75 it/sec]
        INFO - 20:35:08:     33%|███▎      | 33/100 [00:00<00:00, 4764.94 it/sec]
        INFO - 20:35:08:     34%|███▍      | 34/100 [00:00<00:00, 4764.66 it/sec]
        INFO - 20:35:08:     35%|███▌      | 35/100 [00:00<00:00, 4774.16 it/sec]
        INFO - 20:35:08:     36%|███▌      | 36/100 [00:00<00:00, 4783.47 it/sec]
        INFO - 20:35:08:     37%|███▋      | 37/100 [00:00<00:00, 4794.97 it/sec]
        INFO - 20:35:08:     38%|███▊      | 38/100 [00:00<00:00, 4794.79 it/sec]
        INFO - 20:35:08:     39%|███▉      | 39/100 [00:00<00:00, 4797.00 it/sec]
        INFO - 20:35:08:     40%|████      | 40/100 [00:00<00:00, 4804.47 it/sec]
        INFO - 20:35:08:     41%|████      | 41/100 [00:00<00:00, 4785.75 it/sec]
        INFO - 20:35:08:     42%|████▏     | 42/100 [00:00<00:00, 4788.41 it/sec]
        INFO - 20:35:08:     43%|████▎     | 43/100 [00:00<00:00, 4784.59 it/sec]
        INFO - 20:35:08:     44%|████▍     | 44/100 [00:00<00:00, 4788.76 it/sec]
        INFO - 20:35:08:     45%|████▌     | 45/100 [00:00<00:00, 4795.56 it/sec]
        INFO - 20:35:08:     46%|████▌     | 46/100 [00:00<00:00, 4803.63 it/sec]
        INFO - 20:35:08:     47%|████▋     | 47/100 [00:00<00:00, 4810.10 it/sec]
        INFO - 20:35:08:     48%|████▊     | 48/100 [00:00<00:00, 4808.60 it/sec]
        INFO - 20:35:08:     49%|████▉     | 49/100 [00:00<00:00, 4811.45 it/sec]
        INFO - 20:35:08:     50%|█████     | 50/100 [00:00<00:00, 4818.71 it/sec]
        INFO - 20:35:08:     51%|█████     | 51/100 [00:00<00:00, 4824.63 it/sec]
        INFO - 20:35:08:     52%|█████▏    | 52/100 [00:00<00:00, 4830.11 it/sec]
        INFO - 20:35:08:     53%|█████▎    | 53/100 [00:00<00:00, 4825.33 it/sec]
        INFO - 20:35:08:     54%|█████▍    | 54/100 [00:00<00:00, 4829.57 it/sec]
        INFO - 20:35:08:     55%|█████▌    | 55/100 [00:00<00:00, 4835.90 it/sec]
        INFO - 20:35:08:     56%|█████▌    | 56/100 [00:00<00:00, 4840.91 it/sec]
        INFO - 20:35:08:     57%|█████▋    | 57/100 [00:00<00:00, 4845.86 it/sec]
        INFO - 20:35:08:     58%|█████▊    | 58/100 [00:00<00:00, 4845.24 it/sec]
        INFO - 20:35:08:     59%|█████▉    | 59/100 [00:00<00:00, 4848.91 it/sec]
        INFO - 20:35:08:     60%|██████    | 60/100 [00:00<00:00, 4854.52 it/sec]
        INFO - 20:35:08:     61%|██████    | 61/100 [00:00<00:00, 4821.58 it/sec]
        INFO - 20:35:08:     62%|██████▏   | 62/100 [00:00<00:00, 4823.45 it/sec]
        INFO - 20:35:08:     63%|██████▎   | 63/100 [00:00<00:00, 4818.93 it/sec]
        INFO - 20:35:08:     64%|██████▍   | 64/100 [00:00<00:00, 4822.42 it/sec]
        INFO - 20:35:08:     65%|██████▌   | 65/100 [00:00<00:00, 4825.99 it/sec]
        INFO - 20:35:08:     66%|██████▌   | 66/100 [00:00<00:00, 4828.27 it/sec]
        INFO - 20:35:08:     67%|██████▋   | 67/100 [00:00<00:00, 4831.98 it/sec]
        INFO - 20:35:08:     68%|██████▊   | 68/100 [00:00<00:00, 4829.12 it/sec]
        INFO - 20:35:08:     69%|██████▉   | 69/100 [00:00<00:00, 4831.82 it/sec]
        INFO - 20:35:08:     70%|███████   | 70/100 [00:00<00:00, 4834.06 it/sec]
        INFO - 20:35:08:     71%|███████   | 71/100 [00:00<00:00, 4836.46 it/sec]
        INFO - 20:35:08:     72%|███████▏  | 72/100 [00:00<00:00, 4837.26 it/sec]
        INFO - 20:35:08:     73%|███████▎  | 73/100 [00:00<00:00, 4825.83 it/sec]
        INFO - 20:35:08:     74%|███████▍  | 74/100 [00:00<00:00, 4827.94 it/sec]
        INFO - 20:35:08:     75%|███████▌  | 75/100 [00:00<00:00, 4820.00 it/sec]
        INFO - 20:35:08:     76%|███████▌  | 76/100 [00:00<00:00, 4821.33 it/sec]
        INFO - 20:35:08:     77%|███████▋  | 77/100 [00:00<00:00, 4818.88 it/sec]
        INFO - 20:35:08:     78%|███████▊  | 78/100 [00:00<00:00, 4820.33 it/sec]
        INFO - 20:35:08:     79%|███████▉  | 79/100 [00:00<00:00, 4823.43 it/sec]
        INFO - 20:35:08:     80%|████████  | 80/100 [00:00<00:00, 4826.93 it/sec]
        INFO - 20:35:08:     81%|████████  | 81/100 [00:00<00:00, 4831.25 it/sec]
        INFO - 20:35:08:     82%|████████▏ | 82/100 [00:00<00:00, 4830.25 it/sec]
        INFO - 20:35:08:     83%|████████▎ | 83/100 [00:00<00:00, 4833.22 it/sec]
        INFO - 20:35:08:     84%|████████▍ | 84/100 [00:00<00:00, 4837.19 it/sec]
        INFO - 20:35:08:     85%|████████▌ | 85/100 [00:00<00:00, 4840.61 it/sec]
        INFO - 20:35:08:     86%|████████▌ | 86/100 [00:00<00:00, 4842.98 it/sec]
        INFO - 20:35:08:     87%|████████▋ | 87/100 [00:00<00:00, 4841.38 it/sec]
        INFO - 20:35:08:     88%|████████▊ | 88/100 [00:00<00:00, 4843.18 it/sec]
        INFO - 20:35:08:     89%|████████▉ | 89/100 [00:00<00:00, 4846.83 it/sec]
        INFO - 20:35:08:     90%|█████████ | 90/100 [00:00<00:00, 4850.46 it/sec]
        INFO - 20:35:08:     91%|█████████ | 91/100 [00:00<00:00, 4853.41 it/sec]
        INFO - 20:35:08:     92%|█████████▏| 92/100 [00:00<00:00, 4851.28 it/sec]
        INFO - 20:35:08:     93%|█████████▎| 93/100 [00:00<00:00, 4852.77 it/sec]
        INFO - 20:35:08:     94%|█████████▍| 94/100 [00:00<00:00, 4855.83 it/sec]
        INFO - 20:35:08:     95%|█████████▌| 95/100 [00:00<00:00, 4859.25 it/sec]
        INFO - 20:35:08:     96%|█████████▌| 96/100 [00:00<00:00, 4862.90 it/sec]
        INFO - 20:35:08:     97%|█████████▋| 97/100 [00:00<00:00, 4861.42 it/sec]
        INFO - 20:35:08:     98%|█████████▊| 98/100 [00:00<00:00, 4863.59 it/sec]
        INFO - 20:35:08:     99%|█████████▉| 99/100 [00:00<00:00, 4866.69 it/sec]
        INFO - 20:35:08:    100%|██████████| 100/100 [00:00<00:00, 4869.96 it/sec]
        INFO - 20:35:08: *** End Sampling execution ***




.. GENERATED FROM PYTHON SOURCE LINES 111-122

Settings
--------
The :class:`.LinearRegressor` has many options
defined in the :class:`.LinearRegressor_Settings` Pydantic model.

Intercept
~~~~~~~~~~
By default,
the linear model is of the form :math:`a_0+a_1x_1+\ldots+a_dx_d`.
You can set the option ``fit_intercept`` to ``False``
if you want a linear model of the form :math:`a_1x_1+\ldots+a_dx_d`:

.. GENERATED FROM PYTHON SOURCE LINES 122-126

.. code-block:: Python

    model = create_regression_model(
        "LinearRegressor", training_dataset, fit_intercept=False, transformer={}
    )
    model.learn()







.. GENERATED FROM PYTHON SOURCE LINES 127-133

.. warning::
   This notion applies in the space of transformed variables.
   This is the reason why
   we removed the default transformers by setting ``transformer`` to ``{}``.

We can see the impact of this option in the following visualization:

.. GENERATED FROM PYTHON SOURCE LINES 133-141

.. code-block:: Python

    predicted_output_data_ = model.predict(input_data).ravel()
    plt.plot(input_data.ravel(), reference_output_data, label="Reference")
    plt.plot(input_data.ravel(), predicted_output_data, label="Regression - Basics")
    plt.plot(input_data.ravel(), predicted_output_data_, label="Regression - No intercept")
    plt.grid()
    plt.legend()
    plt.show()




.. image-sg:: /examples/mlearning/regression_model/images/sphx_glr_plot_linear_regression_002.png
   :alt: plot linear regression
   :srcset: /examples/mlearning/regression_model/images/sphx_glr_plot_linear_regression_002.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 142-160

Regularization
~~~~~~~~~~~~~~
When the number of samples is small relative to the input dimension,
regularization techniques can save you from overfitting
(a model that is very good at learning but bad at generalization).
The ``penalty_level`` option is a positive real number
defining the degree of regularization (default: no regularization).
By default,
the regularization technique is the ridge penalty (l2 regularization).
The technique can be replaced by the lasso penalty (l1 regularization)
by setting the ``l2_penalty_ratio`` option to ``0.0``.
When ``l2_penalty_ratio`` is between 0 and 1,
the regularization technique is the elastic net penalty,
*i.e.* a linear combination of ridge and lasso penalty
parametrized by this ``l2_penalty_ratio``.

For example,
we can use the ridge penalty with a level of 1.2

.. GENERATED FROM PYTHON SOURCE LINES 160-169

.. code-block:: Python

    model = create_regression_model("LinearRegressor", training_dataset, penalty_level=1.2)
    model.learn()
    predicted_output_data_ = model.predict(input_data).ravel()
    plt.plot(input_data.ravel(), reference_output_data, label="Reference")
    plt.plot(input_data.ravel(), predicted_output_data, label="Regression - Basics")
    plt.plot(input_data.ravel(), predicted_output_data_, label="Regression - Ridge(1.2)")
    plt.grid()
    plt.legend()
    plt.show()



.. image-sg:: /examples/mlearning/regression_model/images/sphx_glr_plot_linear_regression_003.png
   :alt: plot linear regression
   :srcset: /examples/mlearning/regression_model/images/sphx_glr_plot_linear_regression_003.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 170-176

We can see that the coefficient of the linear model is lower due to the penalty.

.. note::
   In the case of a model with many inputs,
   we could have used the lasso penalty
   and seen that some coefficients would have been set to zero.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.204 seconds)


.. _sphx_glr_download_examples_mlearning_regression_model_plot_linear_regression.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_linear_regression.ipynb <plot_linear_regression.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_linear_regression.py <plot_linear_regression.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_linear_regression.zip <plot_linear_regression.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_