Random forest

A RandomForestRegressor is a random forest model based on scikit-learn.

from __future__ import annotations

from matplotlib import pyplot as plt
from numpy import array

from gemseo import create_design_space
from gemseo import create_discipline
from gemseo import sample_disciplines
from gemseo.mlearning import create_regression_model

Problem

In this example, we represent the function \(f(x)=(6x-2)^2\sin(12x-4)\) [FSK08] by the AnalyticDiscipline

discipline = create_discipline(
    "AnalyticDiscipline",
    name="f",
    expressions={"y": "(6*x-2)**2*sin(12*x-4)"},
)

and seek to approximate it over the input space

input_space = create_design_space()
input_space.add_variable("x", lower_bound=0.0, upper_bound=1.0)

To do this, we create a training dataset with 6 equispaced points:

training_dataset = sample_disciplines(
    [discipline], input_space, "y", algo_name="PYDOE_FULLFACT", n_samples=6
)

INFO - 16:19:15: *** Start Sampling execution ***
INFO - 16:19:15: Sampling
INFO - 16:19:15:    Disciplines: f
INFO - 16:19:15:    MDO formulation: MDF
INFO - 16:19:15: Running the algorithm PYDOE_FULLFACT:
INFO - 16:19:15:     17%|█▋        | 1/6 [00:00<00:00, 554.80 it/sec]
INFO - 16:19:15:     33%|███▎      | 2/6 [00:00<00:00, 842.48 it/sec]
INFO - 16:19:15:     50%|█████     | 3/6 [00:00<00:00, 1053.67 it/sec]
INFO - 16:19:15:     67%|██████▋   | 4/6 [00:00<00:00, 1193.94 it/sec]
INFO - 16:19:15:     83%|████████▎ | 5/6 [00:00<00:00, 1307.37 it/sec]
INFO - 16:19:15:    100%|██████████| 6/6 [00:00<00:00, 1362.60 it/sec]
INFO - 16:19:15: *** End Sampling execution ***

Basics

Training

Then, we train an random forest regression model from these samples:

model = create_regression_model("RandomForestRegressor", training_dataset)
model.learn()

Prediction

Once it is built, we can predict the output value of \(f\) at a new input point:

input_value = {"x": array([0.65])}
output_value = model.predict(input_value)
output_value

{'y': array([-0.88837697])}

but cannot predict its Jacobian value:

try:
    model.predict_jacobian(input_value)
except NotImplementedError:
    print("The derivatives are not available for RandomForestRegressor.")

The derivatives are not available for RandomForestRegressor.

Plotting

You can see that the random forest model is pretty good on the left, but bad on the right:

test_dataset = sample_disciplines(
    [discipline], input_space, "y", algo_name="PYDOE_FULLFACT", n_samples=100
)
input_data = test_dataset.get_view(variable_names=model.input_names).to_numpy()
reference_output_data = test_dataset.get_view(variable_names="y").to_numpy().ravel()
predicted_output_data = model.predict(input_data).ravel()
plt.plot(input_data.ravel(), reference_output_data, label="Reference")
plt.plot(input_data.ravel(), predicted_output_data, label="Regression - Basics")
plt.grid()
plt.legend()
plt.show()

INFO - 16:19:15: *** Start Sampling execution ***
INFO - 16:19:15: Sampling
INFO - 16:19:15:    Disciplines: f
INFO - 16:19:15:    MDO formulation: MDF
INFO - 16:19:15: Running the algorithm PYDOE_FULLFACT:
INFO - 16:19:15:      1%|          | 1/100 [00:00<00:00, 2908.67 it/sec]
INFO - 16:19:15:      2%|▏         | 2/100 [00:00<00:00, 2244.74 it/sec]
INFO - 16:19:15:      3%|▎         | 3/100 [00:00<00:00, 2179.99 it/sec]
INFO - 16:19:15:      4%|▍         | 4/100 [00:00<00:00, 2144.33 it/sec]
INFO - 16:19:15:      5%|▌         | 5/100 [00:00<00:00, 2127.36 it/sec]
INFO - 16:19:15:      6%|▌         | 6/100 [00:00<00:00, 2132.70 it/sec]
INFO - 16:19:15:      7%|▋         | 7/100 [00:00<00:00, 2096.25 it/sec]
INFO - 16:19:15:      8%|▊         | 8/100 [00:00<00:00, 2108.62 it/sec]
INFO - 16:19:15:      9%|▉         | 9/100 [00:00<00:00, 2086.03 it/sec]
INFO - 16:19:15:     10%|█         | 10/100 [00:00<00:00, 2097.89 it/sec]
INFO - 16:19:15:     11%|█         | 11/100 [00:00<00:00, 2087.85 it/sec]
INFO - 16:19:15:     12%|█▏        | 12/100 [00:00<00:00, 2097.68 it/sec]
INFO - 16:19:15:     13%|█▎        | 13/100 [00:00<00:00, 2093.85 it/sec]
INFO - 16:19:15:     14%|█▍        | 14/100 [00:00<00:00, 2102.86 it/sec]
INFO - 16:19:15:     15%|█▌        | 15/100 [00:00<00:00, 2102.06 it/sec]
INFO - 16:19:15:     16%|█▌        | 16/100 [00:00<00:00, 2100.96 it/sec]
INFO - 16:19:15:     17%|█▋        | 17/100 [00:00<00:00, 2096.72 it/sec]
INFO - 16:19:15:     18%|█▊        | 18/100 [00:00<00:00, 2104.17 it/sec]
INFO - 16:19:15:     19%|█▉        | 19/100 [00:00<00:00, 2103.46 it/sec]
INFO - 16:19:15:     20%|██        | 20/100 [00:00<00:00, 2103.52 it/sec]
INFO - 16:19:15:     21%|██        | 21/100 [00:00<00:00, 2101.15 it/sec]
INFO - 16:19:15:     22%|██▏       | 22/100 [00:00<00:00, 2104.76 it/sec]
INFO - 16:19:15:     23%|██▎       | 23/100 [00:00<00:00, 2096.29 it/sec]
INFO - 16:19:15:     24%|██▍       | 24/100 [00:00<00:00, 2101.57 it/sec]
INFO - 16:19:15:     25%|██▌       | 25/100 [00:00<00:00, 2093.63 it/sec]
INFO - 16:19:15:     26%|██▌       | 26/100 [00:00<00:00, 2098.65 it/sec]
INFO - 16:19:15:     27%|██▋       | 27/100 [00:00<00:00, 2094.44 it/sec]
INFO - 16:19:15:     28%|██▊       | 28/100 [00:00<00:00, 2099.59 it/sec]
INFO - 16:19:15:     29%|██▉       | 29/100 [00:00<00:00, 2100.81 it/sec]
INFO - 16:19:15:     30%|███       | 30/100 [00:00<00:00, 2100.48 it/sec]
INFO - 16:19:15:     31%|███       | 31/100 [00:00<00:00, 2101.39 it/sec]
INFO - 16:19:15:     32%|███▏      | 32/100 [00:00<00:00, 2083.80 it/sec]
INFO - 16:19:15:     33%|███▎      | 33/100 [00:00<00:00, 2079.23 it/sec]
INFO - 16:19:15:     34%|███▍      | 34/100 [00:00<00:00, 2080.69 it/sec]
INFO - 16:19:15:     35%|███▌      | 35/100 [00:00<00:00, 2081.72 it/sec]
INFO - 16:19:15:     36%|███▌      | 36/100 [00:00<00:00, 2081.86 it/sec]
INFO - 16:19:15:     37%|███▋      | 37/100 [00:00<00:00, 2082.94 it/sec]
INFO - 16:19:15:     38%|███▊      | 38/100 [00:00<00:00, 2083.34 it/sec]
INFO - 16:19:15:     39%|███▉      | 39/100 [00:00<00:00, 2084.38 it/sec]
INFO - 16:19:15:     40%|████      | 40/100 [00:00<00:00, 2085.91 it/sec]
INFO - 16:19:15:     41%|████      | 41/100 [00:00<00:00, 2086.41 it/sec]
INFO - 16:19:15:     42%|████▏     | 42/100 [00:00<00:00, 2085.36 it/sec]
INFO - 16:19:15:     43%|████▎     | 43/100 [00:00<00:00, 2088.14 it/sec]
INFO - 16:19:15:     44%|████▍     | 44/100 [00:00<00:00, 2085.85 it/sec]
INFO - 16:19:15:     45%|████▌     | 45/100 [00:00<00:00, 2088.75 it/sec]
INFO - 16:19:15:     46%|████▌     | 46/100 [00:00<00:00, 2088.14 it/sec]
INFO - 16:19:15:     47%|████▋     | 47/100 [00:00<00:00, 2089.08 it/sec]
INFO - 16:19:15:     48%|████▊     | 48/100 [00:00<00:00, 2088.08 it/sec]
INFO - 16:19:15:     49%|████▉     | 49/100 [00:00<00:00, 2091.26 it/sec]
INFO - 16:19:15:     50%|█████     | 50/100 [00:00<00:00, 2086.84 it/sec]
INFO - 16:19:15:     51%|█████     | 51/100 [00:00<00:00, 2089.31 it/sec]
INFO - 16:19:15:     52%|█████▏    | 52/100 [00:00<00:00, 2086.36 it/sec]
INFO - 16:19:15:     53%|█████▎    | 53/100 [00:00<00:00, 2088.76 it/sec]
INFO - 16:19:15:     54%|█████▍    | 54/100 [00:00<00:00, 2087.05 it/sec]
INFO - 16:19:15:     55%|█████▌    | 55/100 [00:00<00:00, 2089.71 it/sec]
INFO - 16:19:15:     56%|█████▌    | 56/100 [00:00<00:00, 2089.95 it/sec]
INFO - 16:19:15:     57%|█████▋    | 57/100 [00:00<00:00, 2090.82 it/sec]
INFO - 16:19:15:     58%|█████▊    | 58/100 [00:00<00:00, 2091.17 it/sec]
INFO - 16:19:15:     59%|█████▉    | 59/100 [00:00<00:00, 2091.57 it/sec]
INFO - 16:19:15:     60%|██████    | 60/100 [00:00<00:00, 2091.73 it/sec]
INFO - 16:19:15:     61%|██████    | 61/100 [00:00<00:00, 2092.93 it/sec]
INFO - 16:19:15:     62%|██████▏   | 62/100 [00:00<00:00, 2093.13 it/sec]
INFO - 16:19:15:     63%|██████▎   | 63/100 [00:00<00:00, 2093.63 it/sec]
INFO - 16:19:15:     64%|██████▍   | 64/100 [00:00<00:00, 2095.81 it/sec]
INFO - 16:19:15:     65%|██████▌   | 65/100 [00:00<00:00, 2095.59 it/sec]
INFO - 16:19:15:     66%|██████▌   | 66/100 [00:00<00:00, 2096.82 it/sec]
INFO - 16:19:15:     67%|██████▋   | 67/100 [00:00<00:00, 2094.53 it/sec]
INFO - 16:19:15:     68%|██████▊   | 68/100 [00:00<00:00, 2096.35 it/sec]
INFO - 16:19:15:     69%|██████▉   | 69/100 [00:00<00:00, 2094.21 it/sec]
INFO - 16:19:15:     70%|███████   | 70/100 [00:00<00:00, 2096.03 it/sec]
INFO - 16:19:15:     71%|███████   | 71/100 [00:00<00:00, 2094.60 it/sec]
INFO - 16:19:15:     72%|███████▏  | 72/100 [00:00<00:00, 2096.34 it/sec]
INFO - 16:19:15:     73%|███████▎  | 73/100 [00:00<00:00, 2096.33 it/sec]
INFO - 16:19:15:     74%|███████▍  | 74/100 [00:00<00:00, 2096.36 it/sec]
INFO - 16:19:15:     75%|███████▌  | 75/100 [00:00<00:00, 2097.10 it/sec]
INFO - 16:19:15:     76%|███████▌  | 76/100 [00:00<00:00, 2096.75 it/sec]
INFO - 16:19:15:     77%|███████▋  | 77/100 [00:00<00:00, 2096.91 it/sec]
INFO - 16:19:15:     78%|███████▊  | 78/100 [00:00<00:00, 2093.34 it/sec]
INFO - 16:19:15:     79%|███████▉  | 79/100 [00:00<00:00, 2086.99 it/sec]
INFO - 16:19:15:     80%|████████  | 80/100 [00:00<00:00, 2085.36 it/sec]
INFO - 16:19:15:     81%|████████  | 81/100 [00:00<00:00, 2084.61 it/sec]
INFO - 16:19:15:     82%|████████▏ | 82/100 [00:00<00:00, 2085.63 it/sec]
INFO - 16:19:15:     83%|████████▎ | 83/100 [00:00<00:00, 2085.76 it/sec]
INFO - 16:19:15:     84%|████████▍ | 84/100 [00:00<00:00, 2084.84 it/sec]
INFO - 16:19:15:     85%|████████▌ | 85/100 [00:00<00:00, 2084.84 it/sec]
INFO - 16:19:15:     86%|████████▌ | 86/100 [00:00<00:00, 2084.13 it/sec]
INFO - 16:19:15:     87%|████████▋ | 87/100 [00:00<00:00, 2084.60 it/sec]
INFO - 16:19:15:     88%|████████▊ | 88/100 [00:00<00:00, 2084.89 it/sec]
INFO - 16:19:15:     89%|████████▉ | 89/100 [00:00<00:00, 2084.62 it/sec]
INFO - 16:19:15:     90%|█████████ | 90/100 [00:00<00:00, 2085.00 it/sec]
INFO - 16:19:15:     91%|█████████ | 91/100 [00:00<00:00, 2085.50 it/sec]
INFO - 16:19:15:     92%|█████████▏| 92/100 [00:00<00:00, 2085.07 it/sec]
INFO - 16:19:15:     93%|█████████▎| 93/100 [00:00<00:00, 2086.61 it/sec]
INFO - 16:19:15:     94%|█████████▍| 94/100 [00:00<00:00, 2085.58 it/sec]
INFO - 16:19:15:     95%|█████████▌| 95/100 [00:00<00:00, 2087.17 it/sec]
INFO - 16:19:15:     96%|█████████▌| 96/100 [00:00<00:00, 2086.96 it/sec]
INFO - 16:19:15:     97%|█████████▋| 97/100 [00:00<00:00, 2088.45 it/sec]
INFO - 16:19:15:     98%|█████████▊| 98/100 [00:00<00:00, 2086.97 it/sec]
INFO - 16:19:15:     99%|█████████▉| 99/100 [00:00<00:00, 2088.44 it/sec]
INFO - 16:19:15:    100%|██████████| 100/100 [00:00<00:00, 2069.54 it/sec]
INFO - 16:19:15: *** End Sampling execution ***

Settings

Number of estimators

The main hyperparameter of random forest regression is the number of trees in the forest (default: 100). Here is a comparison when increasing and decreasing this number:

model = create_regression_model(
    "RandomForestRegressor", training_dataset, n_estimators=10
)
model.learn()
predicted_output_data_1 = model.predict(input_data).ravel()
model = create_regression_model(
    "RandomForestRegressor", training_dataset, n_estimators=1000
)
model.learn()
predicted_output_data_2 = model.predict(input_data).ravel()
plt.plot(input_data.ravel(), reference_output_data, label="Reference")
plt.plot(input_data.ravel(), predicted_output_data, label="Regression - Basics")
plt.plot(input_data.ravel(), predicted_output_data_1, label="Regression - 10 trees")
plt.plot(input_data.ravel(), predicted_output_data_2, label="Regression - 1000 trees")
plt.grid()
plt.legend()
plt.show()

Others

The RandomForestRegressor class of scikit-learn has a lot of settings (read more), and we have chosen to exhibit only n_estimators. However, any argument of RandomForestRegressor can be set using the dictionary parameters. For example, we can impose a minimum of two samples per leaf:

model = create_regression_model(
    "RandomForestRegressor", training_dataset, parameters={"min_samples_leaf": 2}
)
model.learn()
predicted_output_data_ = model.predict(input_data).ravel()
plt.plot(input_data.ravel(), reference_output_data, label="Reference")
plt.plot(input_data.ravel(), predicted_output_data, label="Regression - Basics")
plt.plot(input_data.ravel(), predicted_output_data_, label="Regression - 2 samples")
plt.grid()
plt.legend()
plt.show()