Note

Go to the end to download the full example code.

Calibration of a polynomial regression#

from __future__ import annotations

import matplotlib.pyplot as plt
from matplotlib.tri import Triangulation

from gemseo.algos.design_space import DesignSpace
from gemseo.mlearning.core.calibration import MLAlgoCalibration
from gemseo.mlearning.regression.quality.mse_measure import MSEMeasure
from gemseo.problems.dataset.rosenbrock import create_rosenbrock_dataset

Load the dataset#

dataset = create_rosenbrock_dataset(opt_naming=False, n_samples=25)

Define the measure#

test_dataset = create_rosenbrock_dataset(opt_naming=False)
measure_evaluation_method_name = "TEST"
measure_options = {"test_data": test_dataset}

Calibrate the degree of the polynomial regression#

Define and execute the calibration#

calibration_space = DesignSpace()
calibration_space.add_variable("degree", 1, "integer", 1, 10, 1)
calibration = MLAlgoCalibration(
    "PolynomialRegressor",
    dataset,
    ["degree"],
    calibration_space,
    MSEMeasure,
    measure_evaluation_method_name=measure_evaluation_method_name,
    measure_options=measure_options,
)
calibration.execute(algo_name="PYDOE_FULLFACT", n_samples=10)
x_opt = calibration.optimal_parameters
f_opt = calibration.optimal_criterion
degree = x_opt["degree"][0]
f"optimal degree = {degree}; optimal criterion = {f_opt}"

    INFO - 16:18:39: *** Start DOEScenario execution ***
    INFO - 16:18:39: DOEScenario
    INFO - 16:18:39:    Disciplines: MLAlgoAssessor
    INFO - 16:18:39:    MDO formulation: DisciplinaryOpt
    INFO - 16:18:39: Optimization problem:
    INFO - 16:18:39:    minimize criterion(degree)
    INFO - 16:18:39:    with respect to degree
    INFO - 16:18:39:    over the design space:
    INFO - 16:18:39:       +--------+-------------+-------+-------------+---------+
    INFO - 16:18:39:       | Name   | Lower bound | Value | Upper bound | Type    |
    INFO - 16:18:39:       +--------+-------------+-------+-------------+---------+
    INFO - 16:18:39:       | degree |      1      |   1   |      10     | integer |
    INFO - 16:18:39:       +--------+-------------+-------+-------------+---------+
    INFO - 16:18:39: Solving optimization problem with algorithm PYDOE_FULLFACT:
    INFO - 16:18:39:     10%|█         | 1/10 [00:00<00:00, 15.45 it/sec, feas=True, obj=5.89e+5]
    INFO - 16:18:39:     20%|██        | 2/10 [00:00<00:00, 26.93 it/sec, feas=True, obj=1.73e+5]
    INFO - 16:18:39:     30%|███       | 3/10 [00:00<00:00, 36.07 it/sec, feas=True, obj=3e+4]
    INFO - 16:18:39:     40%|████      | 4/10 [00:00<00:00, 43.24 it/sec, feas=True, obj=4.23e-24]
    INFO - 16:18:39:     50%|█████     | 5/10 [00:00<00:00, 48.99 it/sec, feas=True, obj=0.11]
    INFO - 16:18:39:     60%|██████    | 6/10 [00:00<00:00, 53.57 it/sec, feas=True, obj=1.18e+3]
    INFO - 16:18:39:     70%|███████   | 7/10 [00:00<00:00, 57.54 it/sec, feas=True, obj=6.9e+3]
    INFO - 16:18:39:     80%|████████  | 8/10 [00:00<00:00, 61.00 it/sec, feas=True, obj=1.36e+4]
    INFO - 16:18:39:     90%|█████████ | 9/10 [00:00<00:00, 64.25 it/sec, feas=True, obj=9.18e+4]
    INFO - 16:18:39:    100%|██████████| 10/10 [00:00<00:00, 67.04 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:39: Optimization result:
    INFO - 16:18:39:    Optimizer info:
    INFO - 16:18:39:       Status: None
    INFO - 16:18:39:       Message: None
    INFO - 16:18:39:    Solution:
    INFO - 16:18:39:       Objective: 4.229341243288758e-24
    INFO - 16:18:39:       Design space:
    INFO - 16:18:39:          +--------+-------------+-------+-------------+---------+
    INFO - 16:18:39:          | Name   | Lower bound | Value | Upper bound | Type    |
    INFO - 16:18:39:          +--------+-------------+-------+-------------+---------+
    INFO - 16:18:39:          | degree |      1      |   4   |      10     | integer |
    INFO - 16:18:39:          +--------+-------------+-------+-------------+---------+
    INFO - 16:18:39: *** End DOEScenario execution ***

'optimal degree = 4; optimal criterion = 4.229341243288758e-24'

Get the history#

calibration.dataset

GROUP	inputs	outputs
VARIABLE	degree	criterion	learning
COMPONENT	0	0	0
0	1	5.888317e+05	8.200828e+05
1	2	1.732475e+05	2.404571e+05
2	3	3.001292e+04	1.645714e+04
3	4	4.229341e-24	6.815305e-24
4	5	1.097877e-01	2.858856e-23
5	6	1.183264e+03	4.777282e-23
6	7	6.895919e+03	5.086705e-24
7	8	1.356307e+04	1.559324e-23
8	9	9.180547e+04	1.399011e-22
9	10	1.625259e+05	7.640253e-23

Visualize the results#

degree = calibration.get_history("degree")
criterion = calibration.get_history("criterion")
learning = calibration.get_history("learning")

plt.plot(degree, criterion, "-o", label="test", color="red")
plt.plot(degree, learning, "-o", label="learning", color="blue")
plt.xlabel("polynomial degree")
plt.ylabel("quality")
plt.axvline(x_opt["degree"], color="red", ls="--")
plt.legend()
plt.show()

plot calibration

Calibrate the ridge penalty of the polynomial regression#

Define and execute the calibration#

calibration_space = DesignSpace()
calibration_space.add_variable("penalty_level", 1, "float", 0.0, 100.0, 0.0)
calibration = MLAlgoCalibration(
    "PolynomialRegressor",
    dataset,
    ["penalty_level"],
    calibration_space,
    MSEMeasure,
    measure_evaluation_method_name=measure_evaluation_method_name,
    measure_options=measure_options,
    degree=10,
)
calibration.execute(algo_name="PYDOE_FULLFACT", n_samples=10)
x_opt = calibration.optimal_parameters
f_opt = calibration.optimal_criterion
x_opt["penalty_level"][0], f_opt

    INFO - 16:18:40: *** Start DOEScenario execution ***
    INFO - 16:18:40: DOEScenario
    INFO - 16:18:40:    Disciplines: MLAlgoAssessor
    INFO - 16:18:40:    MDO formulation: DisciplinaryOpt
    INFO - 16:18:40: Optimization problem:
    INFO - 16:18:40:    minimize criterion(penalty_level)
    INFO - 16:18:40:    with respect to penalty_level
    INFO - 16:18:40:    over the design space:
    INFO - 16:18:40:       +---------------+-------------+-------+-------------+-------+
    INFO - 16:18:40:       | Name          | Lower bound | Value | Upper bound | Type  |
    INFO - 16:18:40:       +---------------+-------------+-------+-------------+-------+
    INFO - 16:18:40:       | penalty_level |      0      |   0   |     100     | float |
    INFO - 16:18:40:       +---------------+-------------+-------+-------------+-------+
    INFO - 16:18:40: Solving optimization problem with algorithm PYDOE_FULLFACT:
    INFO - 16:18:40:     10%|█         | 1/10 [00:00<00:00, 86.23 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:40:     20%|██        | 2/10 [00:00<00:00, 90.96 it/sec, feas=True, obj=3.25e+4]
    INFO - 16:18:40:     30%|███       | 3/10 [00:00<00:00, 95.75 it/sec, feas=True, obj=1.78e+4]
    INFO - 16:18:40:     40%|████      | 4/10 [00:00<00:00, 97.40 it/sec, feas=True, obj=1.72e+4]
    INFO - 16:18:40:     50%|█████     | 5/10 [00:00<00:00, 98.97 it/sec, feas=True, obj=2e+4]
    INFO - 16:18:40:     60%|██████    | 6/10 [00:00<00:00, 99.86 it/sec, feas=True, obj=2.35e+4]
    INFO - 16:18:40:     70%|███████   | 7/10 [00:00<00:00, 101.43 it/sec, feas=True, obj=2.7e+4]
    INFO - 16:18:40:     80%|████████  | 8/10 [00:00<00:00, 102.33 it/sec, feas=True, obj=3.03e+4]
    INFO - 16:18:40:     90%|█████████ | 9/10 [00:00<00:00, 102.83 it/sec, feas=True, obj=3.33e+4]
    INFO - 16:18:40:    100%|██████████| 10/10 [00:00<00:00, 102.83 it/sec, feas=True, obj=3.59e+4]
    INFO - 16:18:40: Optimization result:
    INFO - 16:18:40:    Optimizer info:
    INFO - 16:18:40:       Status: None
    INFO - 16:18:40:       Message: None
    INFO - 16:18:40:    Solution:
    INFO - 16:18:40:       Objective: 17189.526492980985
    INFO - 16:18:40:       Design space:
    INFO - 16:18:40:          +---------------+-------------+-------------------+-------------+-------+
    INFO - 16:18:40:          | Name          | Lower bound |       Value       | Upper bound | Type  |
    INFO - 16:18:40:          +---------------+-------------+-------------------+-------------+-------+
    INFO - 16:18:40:          | penalty_level |      0      | 33.33333333333333 |     100     | float |
    INFO - 16:18:40:          +---------------+-------------+-------------------+-------------+-------+
    INFO - 16:18:40: *** End DOEScenario execution ***

(np.float64(33.33333333333333), np.float64(17189.526492980985))

Get the history#

calibration.dataset

GROUP	inputs	outputs
VARIABLE	penalty_level	criterion	learning
COMPONENT	0	0	0
0	0.000000	162525.860760	7.640253e-23
1	11.111111	32506.221288	1.087801e+03
2	22.222222	17820.599507	1.982580e+03
3	33.333333	17189.526493	2.690007e+03
4	44.444444	19953.420378	3.251453e+03
5	55.555556	23493.269988	3.703714e+03
6	66.666667	27024.053276	4.074147e+03
7	77.777778	30303.486633	4.382362e+03
8	88.888889	33272.062305	4.642448e+03
9	100.000000	35934.745536	4.864667e+03

Visualize the results#

penalty_level = calibration.get_history("penalty_level")
criterion = calibration.get_history("criterion")
learning = calibration.get_history("learning")

plt.plot(penalty_level, criterion, "-o", label="test", color="red")
plt.plot(penalty_level, learning, "-o", label="learning", color="blue")
plt.axvline(x_opt["penalty_level"], color="red", ls="--")
plt.xlabel("ridge penalty")
plt.ylabel("quality")
plt.legend()
plt.show()

plot calibration

Calibrate the lasso penalty of the polynomial regression#

Define and execute the calibration#

calibration_space = DesignSpace()
calibration_space.add_variable("penalty_level", 1, "float", 0.0, 100.0, 0.0)
calibration = MLAlgoCalibration(
    "PolynomialRegressor",
    dataset,
    ["penalty_level"],
    calibration_space,
    MSEMeasure,
    measure_evaluation_method_name=measure_evaluation_method_name,
    measure_options=measure_options,
    degree=10,
    l2_penalty_ratio=0.0,
)
calibration.execute(algo_name="PYDOE_FULLFACT", n_samples=10)
x_opt = calibration.optimal_parameters
f_opt = calibration.optimal_criterion
x_opt["penalty_level"][0], f_opt

    INFO - 16:18:40: *** Start DOEScenario execution ***
    INFO - 16:18:40: DOEScenario
    INFO - 16:18:40:    Disciplines: MLAlgoAssessor
    INFO - 16:18:40:    MDO formulation: DisciplinaryOpt
    INFO - 16:18:40: Optimization problem:
    INFO - 16:18:40:    minimize criterion(penalty_level)
    INFO - 16:18:40:    with respect to penalty_level
    INFO - 16:18:40:    over the design space:
    INFO - 16:18:40:       +---------------+-------------+-------+-------------+-------+
    INFO - 16:18:40:       | Name          | Lower bound | Value | Upper bound | Type  |
    INFO - 16:18:40:       +---------------+-------------+-------+-------------+-------+
    INFO - 16:18:40:       | penalty_level |      0      |   0   |     100     | float |
    INFO - 16:18:40:       +---------------+-------------+-------+-------------+-------+
    INFO - 16:18:40: Solving optimization problem with algorithm PYDOE_FULLFACT:
    INFO - 16:18:40:     10%|█         | 1/10 [00:00<00:00, 81.47 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:40:     20%|██        | 2/10 [00:00<00:00, 82.85 it/sec, feas=True, obj=1.58e+4]
    INFO - 16:18:40:     30%|███       | 3/10 [00:00<00:00, 83.48 it/sec, feas=True, obj=3.15e+4]
    INFO - 16:18:40:     40%|████      | 4/10 [00:00<00:00, 84.86 it/sec, feas=True, obj=4.74e+4]
    INFO - 16:18:40:     50%|█████     | 5/10 [00:00<00:00, 85.86 it/sec, feas=True, obj=5.94e+4]
    INFO - 16:18:40:     60%|██████    | 6/10 [00:00<00:00, 86.73 it/sec, feas=True, obj=6.27e+4]
    INFO - 16:18:40:     70%|███████   | 7/10 [00:00<00:00, 87.41 it/sec, feas=True, obj=6.63e+4]
    INFO - 16:18:40:     80%|████████  | 8/10 [00:00<00:00, 87.77 it/sec, feas=True, obj=6.93e+4]
    INFO - 16:18:40:     90%|█████████ | 9/10 [00:00<00:00, 85.83 it/sec, feas=True, obj=7.25e+4]
    INFO - 16:18:40:    100%|██████████| 10/10 [00:00<00:00, 85.81 it/sec, feas=True, obj=7.57e+4]
    INFO - 16:18:40: Optimization result:
    INFO - 16:18:40:    Optimizer info:
    INFO - 16:18:40:       Status: None
    INFO - 16:18:40:       Message: None
    INFO - 16:18:40:    Solution:
    INFO - 16:18:40:       Objective: 15775.989581125858
    INFO - 16:18:40:       Design space:
    INFO - 16:18:40:          +---------------+-------------+-------------------+-------------+-------+
    INFO - 16:18:40:          | Name          | Lower bound |       Value       | Upper bound | Type  |
    INFO - 16:18:40:          +---------------+-------------+-------------------+-------------+-------+
    INFO - 16:18:40:          | penalty_level |      0      | 11.11111111111111 |     100     | float |
    INFO - 16:18:40:          +---------------+-------------+-------------------+-------------+-------+
    INFO - 16:18:40: *** End DOEScenario execution ***

(np.float64(11.11111111111111), np.float64(15775.989581125858))

Get the history#

calibration.dataset

GROUP	inputs	outputs
VARIABLE	penalty_level	criterion	learning
COMPONENT	0	0	0
0	0.000000	162525.860760	7.640253e-23
1	11.111111	15775.989581	1.814382e+03
2	22.222222	31529.584354	4.057302e+03
3	33.333333	47420.249503	5.792299e+03
4	44.444444	59358.207437	7.169565e+03
5	55.555556	62656.171431	7.278397e+03
6	66.666667	66256.259889	7.410137e+03
7	77.777778	69336.190346	7.540731e+03
8	88.888889	72457.378777	7.675963e+03
9	100.000000	75749.793494	7.816545e+03

Visualize the results#

penalty_level = calibration.get_history("penalty_level")
criterion = calibration.get_history("criterion")
learning = calibration.get_history("learning")

plt.plot(penalty_level, criterion, "-o", label="test", color="red")
plt.plot(penalty_level, learning, "-o", label="learning", color="blue")
plt.axvline(x_opt["penalty_level"], color="red", ls="--")
plt.xlabel("lasso penalty")
plt.ylabel("quality")
plt.legend()
plt.show()

plot calibration

Calibrate the elasticnet penalty of the polynomial regression#

Define and execute the calibration#

calibration_space = DesignSpace()
calibration_space.add_variable("penalty_level", 1, "float", 0.0, 40.0, 0.0)
calibration_space.add_variable("l2_penalty_ratio", 1, "float", 0.0, 1.0, 0.5)
calibration = MLAlgoCalibration(
    "PolynomialRegressor",
    dataset,
    ["penalty_level", "l2_penalty_ratio"],
    calibration_space,
    MSEMeasure,
    measure_evaluation_method_name=measure_evaluation_method_name,
    measure_options=measure_options,
    degree=10,
)
calibration.execute(algo_name="PYDOE_FULLFACT", n_samples=100)
x_opt = calibration.optimal_parameters
f_opt = calibration.optimal_criterion
x_opt["penalty_level"][0], x_opt["l2_penalty_ratio"][0], f_opt

    INFO - 16:18:40: *** Start DOEScenario execution ***
    INFO - 16:18:40: DOEScenario
    INFO - 16:18:40:    Disciplines: MLAlgoAssessor
    INFO - 16:18:40:    MDO formulation: DisciplinaryOpt
    INFO - 16:18:40: Optimization problem:
    INFO - 16:18:40:    minimize criterion(penalty_level, l2_penalty_ratio)
    INFO - 16:18:40:    with respect to l2_penalty_ratio, penalty_level
    INFO - 16:18:40:    over the design space:
    INFO - 16:18:40:       +------------------+-------------+-------+-------------+-------+
    INFO - 16:18:40:       | Name             | Lower bound | Value | Upper bound | Type  |
    INFO - 16:18:40:       +------------------+-------------+-------+-------------+-------+
    INFO - 16:18:40:       | penalty_level    |      0      |   0   |      40     | float |
    INFO - 16:18:40:       | l2_penalty_ratio |      0      |  0.5  |      1      | float |
    INFO - 16:18:40:       +------------------+-------------+-------+-------------+-------+
    INFO - 16:18:40: Solving optimization problem with algorithm PYDOE_FULLFACT:
    INFO - 16:18:40:      1%|          | 1/100 [00:00<00:01, 81.50 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:40:      2%|▏         | 2/100 [00:00<00:01, 83.14 it/sec, feas=True, obj=4.14e+3]
    INFO - 16:18:40:      3%|▎         | 3/100 [00:00<00:01, 84.47 it/sec, feas=True, obj=1.34e+4]
    INFO - 16:18:40:      4%|▍         | 4/100 [00:00<00:01, 84.75 it/sec, feas=True, obj=1.79e+4]
    INFO - 16:18:40:      5%|▌         | 5/100 [00:00<00:01, 85.59 it/sec, feas=True, obj=2.39e+4]
    INFO - 16:18:40:      6%|▌         | 6/100 [00:00<00:01, 86.16 it/sec, feas=True, obj=3.15e+4]
    INFO - 16:18:40:      7%|▋         | 7/100 [00:00<00:01, 86.46 it/sec, feas=True, obj=3.91e+4]
    INFO - 16:18:40:      8%|▊         | 8/100 [00:00<00:01, 86.90 it/sec, feas=True, obj=4.5e+4]
    INFO - 16:18:40:      9%|▉         | 9/100 [00:00<00:01, 87.15 it/sec, feas=True, obj=4.95e+4]
    INFO - 16:18:40:     10%|█         | 10/100 [00:00<00:01, 87.48 it/sec, feas=True, obj=5.42e+4]
    INFO - 16:18:40:     11%|█         | 11/100 [00:00<00:00, 89.30 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:40:     12%|█▏        | 12/100 [00:00<00:00, 89.00 it/sec, feas=True, obj=1.35e+4]
    INFO - 16:18:40:     13%|█▎        | 13/100 [00:00<00:00, 88.84 it/sec, feas=True, obj=2.44e+4]
    INFO - 16:18:40:     14%|█▍        | 14/100 [00:00<00:00, 88.71 it/sec, feas=True, obj=3.28e+4]
    INFO - 16:18:40:     15%|█▌        | 15/100 [00:00<00:00, 88.69 it/sec, feas=True, obj=4.19e+4]
    INFO - 16:18:40:     16%|█▌        | 16/100 [00:00<00:00, 88.62 it/sec, feas=True, obj=4.76e+4]
    INFO - 16:18:40:     17%|█▋        | 17/100 [00:00<00:00, 88.58 it/sec, feas=True, obj=5.16e+4]
    INFO - 16:18:40:     18%|█▊        | 18/100 [00:00<00:00, 88.52 it/sec, feas=True, obj=5.52e+4]
    INFO - 16:18:40:     19%|█▉        | 19/100 [00:00<00:00, 88.55 it/sec, feas=True, obj=5.78e+4]
    INFO - 16:18:40:     20%|██        | 20/100 [00:00<00:00, 88.58 it/sec, feas=True, obj=5.98e+4]
    INFO - 16:18:40:     21%|██        | 21/100 [00:00<00:00, 89.47 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:40:     22%|██▏       | 22/100 [00:00<00:00, 89.20 it/sec, feas=True, obj=1.97e+4]
    INFO - 16:18:40:     23%|██▎       | 23/100 [00:00<00:00, 88.99 it/sec, feas=True, obj=3.17e+4]
    INFO - 16:18:40:     24%|██▍       | 24/100 [00:00<00:00, 88.90 it/sec, feas=True, obj=4.02e+4]
    INFO - 16:18:40:     25%|██▌       | 25/100 [00:00<00:00, 88.74 it/sec, feas=True, obj=4.59e+4]
    INFO - 16:18:40:     26%|██▌       | 26/100 [00:00<00:00, 88.61 it/sec, feas=True, obj=4.97e+4]
    INFO - 16:18:40:     27%|██▋       | 27/100 [00:00<00:00, 88.53 it/sec, feas=True, obj=5.3e+4]
    INFO - 16:18:40:     28%|██▊       | 28/100 [00:00<00:00, 88.43 it/sec, feas=True, obj=5.6e+4]
    INFO - 16:18:40:     29%|██▉       | 29/100 [00:00<00:00, 88.39 it/sec, feas=True, obj=5.89e+4]
    INFO - 16:18:40:     30%|███       | 30/100 [00:00<00:00, 88.32 it/sec, feas=True, obj=6.18e+4]
    INFO - 16:18:40:     31%|███       | 31/100 [00:00<00:00, 88.85 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:40:     32%|███▏      | 32/100 [00:00<00:00, 88.68 it/sec, feas=True, obj=2.43e+4]
    INFO - 16:18:40:     33%|███▎      | 33/100 [00:00<00:00, 88.55 it/sec, feas=True, obj=3.58e+4]
    INFO - 16:18:40:     34%|███▍      | 34/100 [00:00<00:00, 88.41 it/sec, feas=True, obj=4.29e+4]
    INFO - 16:18:40:     35%|███▌      | 35/100 [00:00<00:00, 88.33 it/sec, feas=True, obj=4.77e+4]
    INFO - 16:18:40:     36%|███▌      | 36/100 [00:00<00:00, 88.17 it/sec, feas=True, obj=5.12e+4]
    INFO - 16:18:40:     37%|███▋      | 37/100 [00:00<00:00, 88.12 it/sec, feas=True, obj=5.43e+4]
    INFO - 16:18:40:     38%|███▊      | 38/100 [00:00<00:00, 88.05 it/sec, feas=True, obj=5.74e+4]
    INFO - 16:18:40:     39%|███▉      | 39/100 [00:00<00:00, 87.99 it/sec, feas=True, obj=6.05e+4]
    INFO - 16:18:40:     40%|████      | 40/100 [00:00<00:00, 87.47 it/sec, feas=True, obj=6.35e+4]
    INFO - 16:18:40:     41%|████      | 41/100 [00:00<00:00, 87.90 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:40:     42%|████▏     | 42/100 [00:00<00:00, 87.80 it/sec, feas=True, obj=2.75e+4]
    INFO - 16:18:40:     43%|████▎     | 43/100 [00:00<00:00, 87.73 it/sec, feas=True, obj=3.82e+4]
    INFO - 16:18:40:     44%|████▍     | 44/100 [00:00<00:00, 87.68 it/sec, feas=True, obj=4.42e+4]
    INFO - 16:18:40:     45%|████▌     | 45/100 [00:00<00:00, 87.66 it/sec, feas=True, obj=4.9e+4]
    INFO - 16:18:40:     46%|████▌     | 46/100 [00:00<00:00, 87.66 it/sec, feas=True, obj=5.28e+4]
    INFO - 16:18:40:     47%|████▋     | 47/100 [00:00<00:00, 87.66 it/sec, feas=True, obj=5.61e+4]
    INFO - 16:18:40:     48%|████▊     | 48/100 [00:00<00:00, 87.63 it/sec, feas=True, obj=5.93e+4]
    INFO - 16:18:41:     49%|████▉     | 49/100 [00:00<00:00, 87.59 it/sec, feas=True, obj=6.24e+4]
    INFO - 16:18:41:     50%|█████     | 50/100 [00:00<00:00, 87.52 it/sec, feas=True, obj=6.54e+4]
    INFO - 16:18:41:     51%|█████     | 51/100 [00:00<00:00, 87.83 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:41:     52%|█████▏    | 52/100 [00:00<00:00, 87.61 it/sec, feas=True, obj=2.99e+4]
    INFO - 16:18:41:     53%|█████▎    | 53/100 [00:00<00:00, 87.50 it/sec, feas=True, obj=3.96e+4]
    INFO - 16:18:41:     54%|█████▍    | 54/100 [00:00<00:00, 87.37 it/sec, feas=True, obj=4.51e+4]
    INFO - 16:18:41:     55%|█████▌    | 55/100 [00:00<00:00, 87.29 it/sec, feas=True, obj=5e+4]
    INFO - 16:18:41:     56%|█████▌    | 56/100 [00:00<00:00, 87.17 it/sec, feas=True, obj=5.43e+4]
    INFO - 16:18:41:     57%|█████▋    | 57/100 [00:00<00:00, 87.10 it/sec, feas=True, obj=5.78e+4]
    INFO - 16:18:41:     58%|█████▊    | 58/100 [00:00<00:00, 87.05 it/sec, feas=True, obj=6.11e+4]
    INFO - 16:18:41:     59%|█████▉    | 59/100 [00:00<00:00, 87.01 it/sec, feas=True, obj=6.41e+4]
    INFO - 16:18:41:     60%|██████    | 60/100 [00:00<00:00, 87.01 it/sec, feas=True, obj=6.66e+4]
    INFO - 16:18:41:     61%|██████    | 61/100 [00:00<00:00, 87.34 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:41:     62%|██████▏   | 62/100 [00:00<00:00, 87.30 it/sec, feas=True, obj=3.18e+4]
    INFO - 16:18:41:     63%|██████▎   | 63/100 [00:00<00:00, 87.23 it/sec, feas=True, obj=4.07e+4]
    INFO - 16:18:41:     64%|██████▍   | 64/100 [00:00<00:00, 87.17 it/sec, feas=True, obj=4.6e+4]
    INFO - 16:18:41:     65%|██████▌   | 65/100 [00:00<00:00, 87.09 it/sec, feas=True, obj=5.09e+4]
    INFO - 16:18:41:     66%|██████▌   | 66/100 [00:00<00:00, 87.01 it/sec, feas=True, obj=5.53e+4]
    INFO - 16:18:41:     67%|██████▋   | 67/100 [00:00<00:00, 86.99 it/sec, feas=True, obj=5.92e+4]
    INFO - 16:18:41:     68%|██████▊   | 68/100 [00:00<00:00, 87.00 it/sec, feas=True, obj=6.25e+4]
    INFO - 16:18:41:     69%|██████▉   | 69/100 [00:00<00:00, 87.02 it/sec, feas=True, obj=6.52e+4]
    INFO - 16:18:41:     70%|███████   | 70/100 [00:00<00:00, 86.96 it/sec, feas=True, obj=6.71e+4]
    INFO - 16:18:41:     71%|███████   | 71/100 [00:00<00:00, 87.20 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:41:     72%|███████▏  | 72/100 [00:00<00:00, 87.14 it/sec, feas=True, obj=3.32e+4]
    INFO - 16:18:41:     73%|███████▎  | 73/100 [00:00<00:00, 87.10 it/sec, feas=True, obj=4.15e+4]
    INFO - 16:18:41:     74%|███████▍  | 74/100 [00:00<00:00, 87.03 it/sec, feas=True, obj=4.69e+4]
    INFO - 16:18:41:     75%|███████▌  | 75/100 [00:00<00:00, 86.97 it/sec, feas=True, obj=5.19e+4]
    INFO - 16:18:41:     76%|███████▌  | 76/100 [00:00<00:00, 86.86 it/sec, feas=True, obj=5.63e+4]
    INFO - 16:18:41:     77%|███████▋  | 77/100 [00:00<00:00, 86.75 it/sec, feas=True, obj=6.01e+4]
    INFO - 16:18:41:     78%|███████▊  | 78/100 [00:00<00:00, 86.70 it/sec, feas=True, obj=6.32e+4]
    INFO - 16:18:41:     79%|███████▉  | 79/100 [00:00<00:00, 86.63 it/sec, feas=True, obj=6.55e+4]
    INFO - 16:18:41:     80%|████████  | 80/100 [00:00<00:00, 86.61 it/sec, feas=True, obj=6.72e+4]
    INFO - 16:18:41:     81%|████████  | 81/100 [00:00<00:00, 86.84 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:41:     82%|████████▏ | 82/100 [00:00<00:00, 86.79 it/sec, feas=True, obj=3.44e+4]
    INFO - 16:18:41:     83%|████████▎ | 83/100 [00:00<00:00, 86.79 it/sec, feas=True, obj=4.23e+4]
    INFO - 16:18:41:     84%|████████▍ | 84/100 [00:00<00:00, 86.74 it/sec, feas=True, obj=4.78e+4]
    INFO - 16:18:41:     85%|████████▌ | 85/100 [00:00<00:00, 86.68 it/sec, feas=True, obj=5.3e+4]
    INFO - 16:18:41:     86%|████████▌ | 86/100 [00:00<00:00, 86.66 it/sec, feas=True, obj=5.74e+4]
    INFO - 16:18:41:     87%|████████▋ | 87/100 [00:01<00:00, 86.66 it/sec, feas=True, obj=6.09e+4]
    INFO - 16:18:41:     88%|████████▊ | 88/100 [00:01<00:00, 86.60 it/sec, feas=True, obj=6.35e+4]
    INFO - 16:18:41:     89%|████████▉ | 89/100 [00:01<00:00, 86.58 it/sec, feas=True, obj=6.53e+4]
    INFO - 16:18:41:     90%|█████████ | 90/100 [00:01<00:00, 86.55 it/sec, feas=True, obj=6.67e+4]
    INFO - 16:18:41:     91%|█████████ | 91/100 [00:01<00:00, 86.76 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:41:     92%|█████████▏| 92/100 [00:01<00:00, 86.97 it/sec, feas=True, obj=6.89e+4]
    INFO - 16:18:41:     93%|█████████▎| 93/100 [00:01<00:00, 87.15 it/sec, feas=True, obj=4.03e+4]
    INFO - 16:18:41:     94%|█████████▍| 94/100 [00:01<00:00, 87.36 it/sec, feas=True, obj=2.71e+4]
    INFO - 16:18:41:     95%|█████████▌| 95/100 [00:01<00:00, 87.58 it/sec, feas=True, obj=2.07e+4]
    INFO - 16:18:41:     96%|█████████▌| 96/100 [00:01<00:00, 87.78 it/sec, feas=True, obj=1.78e+4]
    INFO - 16:18:41:     97%|█████████▋| 97/100 [00:01<00:00, 87.96 it/sec, feas=True, obj=1.68e+4]
    INFO - 16:18:41:     98%|█████████▊| 98/100 [00:01<00:00, 88.16 it/sec, feas=True, obj=1.69e+4]
    INFO - 16:18:41:     99%|█████████▉| 99/100 [00:01<00:00, 88.35 it/sec, feas=True, obj=1.76e+4]
    INFO - 16:18:41:    100%|██████████| 100/100 [00:01<00:00, 88.52 it/sec, feas=True, obj=1.87e+4]
    INFO - 16:18:41: Optimization result:
    INFO - 16:18:41:    Optimizer info:
    INFO - 16:18:41:       Status: None
    INFO - 16:18:41:       Message: None
    INFO - 16:18:41:    Solution:
    INFO - 16:18:41:       Objective: 4136.820826715572
    INFO - 16:18:41:       Design space:
    INFO - 16:18:41:          +------------------+-------------+-------------------+-------------+-------+
    INFO - 16:18:41:          | Name             | Lower bound |       Value       | Upper bound | Type  |
    INFO - 16:18:41:          +------------------+-------------+-------------------+-------------+-------+
    INFO - 16:18:41:          | penalty_level    |      0      | 4.444444444444445 |      40     | float |
    INFO - 16:18:41:          | l2_penalty_ratio |      0      |         0         |      1      | float |
    INFO - 16:18:41:          +------------------+-------------+-------------------+-------------+-------+
    INFO - 16:18:41: *** End DOEScenario execution ***

(np.float64(4.444444444444445), np.float64(0.0), np.float64(4136.820826715572))

Get the history#

calibration.dataset

GROUP	inputs		outputs
VARIABLE	penalty_level	l2_penalty_ratio	criterion	learning
COMPONENT	0	0	0	0
0	0.000000	0.0	162525.860760	7.640253e-23
1	4.444444	0.0	4136.820827	4.546714e+02
2	8.888889	0.0	13371.034446	1.375915e+03
3	13.333333	0.0	17860.819693	2.176736e+03
4	17.777778	0.0	23914.366014	3.005032e+03
...	...	...	...	...
95	22.222222	1.0	17820.599507	1.982580e+03
96	26.666667	1.0	16816.595592	2.285780e+03
97	31.111111	1.0	16894.821607	2.561538e+03
98	35.555556	1.0	17602.178769	2.812662e+03
99	40.000000	1.0	18674.751406	3.041823e+03

100 rows × 4 columns

Visualize the results#

penalty_level = calibration.get_history("penalty_level").flatten()
l2_penalty_ratio = calibration.get_history("l2_penalty_ratio").flatten()
criterion = calibration.get_history("criterion").flatten()
learning = calibration.get_history("learning").flatten()

triang = Triangulation(penalty_level, l2_penalty_ratio)

fig = plt.figure()
ax = fig.add_subplot(1, 2, 1)
ax.tricontourf(triang, criterion, cmap="Purples")
ax.scatter(x_opt["penalty_level"][0], x_opt["l2_penalty_ratio"][0])
ax.set_xlabel("penalty level")
ax.set_ylabel("l2 penalty ratio")
ax.set_title("Test measure")
ax = fig.add_subplot(1, 2, 2)
ax.tricontourf(triang, learning, cmap="Purples")
ax.scatter(x_opt["penalty_level"][0], x_opt["l2_penalty_ratio"][0])
ax.set_xlabel("penalty level")
ax.set_ylabel("l2 penalty ratio")
ax.set_title("Learning measure")

plt.show()

Test measure, Learning measure

Add an optimization stage#

calibration_space = DesignSpace()
calibration_space.add_variable("penalty_level", 1, "float", 0.0, 40.0, 0.0)
calibration_space.add_variable("l2_penalty_ratio", 1, "float", 0.0, 1.0, 0.5)
calibration = MLAlgoCalibration(
    "PolynomialRegressor",
    dataset,
    ["penalty_level", "l2_penalty_ratio"],
    calibration_space,
    MSEMeasure,
    measure_evaluation_method_name=measure_evaluation_method_name,
    measure_options=measure_options,
    degree=10,
)
calibration.execute("NLOPT_COBYLA", max_iter=100)
x_opt2 = calibration.optimal_parameters
f_opt2 = calibration.optimal_criterion

fig = plt.figure()
ax = fig.add_subplot(1, 2, 1)
ax.tricontourf(triang, criterion, cmap="Purples")
ax.scatter(x_opt["penalty_level"][0], x_opt["l2_penalty_ratio"][0])
ax.scatter(x_opt2["penalty_level"][0], x_opt2["l2_penalty_ratio"][0], color="red")
ax.set_xlabel("penalty level")
ax.set_ylabel("l2 penalty ratio")
ax.set_title("Test measure")
ax = fig.add_subplot(1, 2, 2)
ax.tricontourf(triang, learning, cmap="Purples")
ax.scatter(x_opt["penalty_level"][0], x_opt["l2_penalty_ratio"][0])
ax.scatter(x_opt2["penalty_level"][0], x_opt2["l2_penalty_ratio"][0], color="red")
ax.set_xlabel("penalty level")
ax.set_ylabel("l2 penalty ratio")
ax.set_title("Learning measure")
plt.show()

n_iterations = len(calibration.scenario.disciplines[0].cache)
print(f"MSE with DOE: {f_opt} (100 evaluations)")
print(f"MSE with OPT: {f_opt2} ({n_iterations} evaluations)")
print(f"MSE reduction:{round((f_opt2 - f_opt) / f_opt * 100)}%")

Test measure, Learning measure

    INFO - 16:18:41: *** Start MDOScenario execution ***
    INFO - 16:18:41: MDOScenario
    INFO - 16:18:41:    Disciplines: MLAlgoAssessor
    INFO - 16:18:41:    MDO formulation: DisciplinaryOpt
    INFO - 16:18:41: Optimization problem:
    INFO - 16:18:41:    minimize criterion(penalty_level, l2_penalty_ratio)
    INFO - 16:18:41:    with respect to l2_penalty_ratio, penalty_level
    INFO - 16:18:41:    over the design space:
    INFO - 16:18:41:       +------------------+-------------+-------+-------------+-------+
    INFO - 16:18:41:       | Name             | Lower bound | Value | Upper bound | Type  |
    INFO - 16:18:41:       +------------------+-------------+-------+-------------+-------+
    INFO - 16:18:41:       | penalty_level    |      0      |   0   |      40     | float |
    INFO - 16:18:41:       | l2_penalty_ratio |      0      |  0.5  |      1      | float |
    INFO - 16:18:41:       +------------------+-------------+-------+-------------+-------+
    INFO - 16:18:41: Solving optimization problem with algorithm NLOPT_COBYLA:
    INFO - 16:18:41:      1%|          | 1/100 [00:00<00:01, 80.25 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:41:      2%|▏         | 2/100 [00:00<00:01, 81.04 it/sec, feas=True, obj=4.06e+4]
    INFO - 16:18:41:      3%|▎         | 3/100 [00:00<00:01, 80.65 it/sec, feas=True, obj=4.28e+4]
    INFO - 16:18:41:      4%|▍         | 4/100 [00:00<00:01, 80.93 it/sec, feas=True, obj=5.16e+4]
    INFO - 16:18:41:      5%|▌         | 5/100 [00:00<00:01, 80.44 it/sec, feas=True, obj=4.66e+4]
    INFO - 16:18:41:      6%|▌         | 6/100 [00:00<00:01, 80.30 it/sec, feas=True, obj=3.63e+4]
    INFO - 16:18:41:      7%|▋         | 7/100 [00:00<00:01, 80.15 it/sec, feas=True, obj=3.03e+4]
    INFO - 16:18:41:      8%|▊         | 8/100 [00:00<00:01, 79.62 it/sec, feas=True, obj=2.03e+4]
    INFO - 16:18:41:      9%|▉         | 9/100 [00:00<00:01, 79.56 it/sec, feas=True, obj=2.75e+3]
    INFO - 16:18:41:     10%|█         | 10/100 [00:00<00:01, 81.34 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:41:     11%|█         | 11/100 [00:00<00:01, 80.93 it/sec, feas=True, obj=3.78e+3]
    INFO - 16:18:41:     12%|█▏        | 12/100 [00:00<00:01, 82.16 it/sec, feas=True, obj=1.63e+5]
    INFO - 16:18:41:     13%|█▎        | 13/100 [00:00<00:01, 81.93 it/sec, feas=True, obj=3.94e+3]
    INFO - 16:18:41:     14%|█▍        | 14/100 [00:00<00:01, 81.68 it/sec, feas=True, obj=4.51e+3]
    INFO - 16:18:41:     15%|█▌        | 15/100 [00:00<00:01, 82.04 it/sec, feas=True, obj=493]
    INFO - 16:18:41:     16%|█▌        | 16/100 [00:00<00:01, 82.56 it/sec, feas=True, obj=493]
    INFO - 16:18:41:     17%|█▋        | 17/100 [00:00<00:00, 83.08 it/sec, feas=True, obj=493]
    INFO - 16:18:41: Optimization result:
    INFO - 16:18:41:    Optimizer info:
    INFO - 16:18:41:       Status: None
    INFO - 16:18:41:       Message: Successive iterates of the objective function are closer than ftol_rel or ftol_abs. GEMSEO stopped the driver.
    INFO - 16:18:41:    Solution:
    INFO - 16:18:41:       Objective: 493.18182004969594
    INFO - 16:18:41:       Design space:
    INFO - 16:18:41:          +------------------+-------------+-----------------------+-------------+-------+
    INFO - 16:18:41:          | Name             | Lower bound |         Value         | Upper bound | Type  |
    INFO - 16:18:41:          +------------------+-------------+-----------------------+-------------+-------+
    INFO - 16:18:41:          | penalty_level    |      0      | 1.191314350140957e-17 |      40     | float |
    INFO - 16:18:41:          | l2_penalty_ratio |      0      |   0.4904563580566946  |      1      | float |
    INFO - 16:18:41:          +------------------+-------------+-----------------------+-------------+-------+
    INFO - 16:18:41: *** End MDOScenario execution ***
MSE with DOE: 4136.820826715572 (100 evaluations)
MSE with OPT: 493.18182004969594 (1 evaluations)
MSE reduction:-88%

Total running time of the script: (0 minutes 2.178 seconds)

Gallery generated by Sphinx-Gallery