Mixture of experts

In this demo, we load a dataset (the Rosenbrock function in 2D) and apply a mixture of experts regression model to obtain an approximation.

from __future__ import annotations

import matplotlib.pyplot as plt
from gemseo.api import configure_logger
from gemseo.api import load_dataset
from gemseo.mlearning.api import create_regression_model
from gemseo.mlearning.transform.scaler.min_max_scaler import MinMaxScaler
from numpy import array
from numpy import hstack
from numpy import linspace
from numpy import meshgrid
from numpy import nonzero
from numpy import sqrt
from numpy import zeros

configure_logger()
<RootLogger root (INFO)>

Dataset (Rosenbrock)

We here consider the Rosenbrock function with two inputs, on the interval \([-2, 2] \times [-2, 2]\).

Load dataset

A prebuilt dataset for the Rosenbrock function with two inputs is given as a dataset parametrization, based on a full factorial DOE of the input space with 100 points.

dataset = load_dataset("RosenbrockDataset", opt_naming=False)

Show dataset

The dataset object can present the data in tabular form.

print(dataset.export_to_dataframe())
        outputs    inputs
          rosen         x
              0         0    1
0   3609.000000 -2.000000 -2.0
1   1959.952599 -1.555556 -2.0
2   1050.699741 -1.111111 -2.0
3    600.308642 -0.666667 -2.0
4    421.490779 -0.222222 -2.0
..          ...       ...  ...
95   381.095717  0.222222  2.0
96   242.086420  0.666667  2.0
97    58.600975  1.111111  2.0
98    17.927907  1.555556  2.0
99   401.000000  2.000000  2.0

[100 rows x 3 columns]

Mixture of experts (MoE)

In this section we load a mixture of experts regression model through the machine learning API, using clustering, classification and regression models.

Mixture of experts model

We construct the MoE model using the predefined parameters, and fit the model to the dataset through the learn() method.

model = create_regression_model(
    "MOERegressor", dataset, transformer={"outputs": MinMaxScaler()}
)
model.set_clusterer("KMeans", n_clusters=3)
model.set_classifier("KNNClassifier", n_neighbors=5)
model.set_regressor("GaussianProcessRegressor")

model.learn()
/home/docs/checkouts/readthedocs.org/user_builds/gemseo/envs/develop/lib/python3.9/site-packages/sklearn/gaussian_process/_gpr.py:616: ConvergenceWarning: lbfgs failed to converge (status=2):
ABNORMAL_TERMINATION_IN_LNSRCH.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
  _check_optimize_result("lbfgs", opt_res)
/home/docs/checkouts/readthedocs.org/user_builds/gemseo/envs/develop/lib/python3.9/site-packages/sklearn/gaussian_process/_gpr.py:616: ConvergenceWarning: lbfgs failed to converge (status=2):
ABNORMAL_TERMINATION_IN_LNSRCH.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
  _check_optimize_result("lbfgs", opt_res)

Tests

Here, we test the mixture of experts method applied to two points: (1, 1), the global minimum, where the function is zero, and (-2, -2), an extreme point where the function has a high value (max on the domain). The classes are expected to be different at the two points.

input_value = {"x": array([1, 1])}
another_input_value = {"x": array([[1, 1], [-2, -2]])}

for value in [input_value, another_input_value]:
    print("Input value:", value)
    print("Class:", model.predict_class(value))
    print("Prediction:", model.predict(value))
    print("Local model predictions:")
    for cls in range(model.n_clusters):
        print(f"Local model {cls}: {model.predict_local_model(value, cls)}")
    print()
Input value: {'x': array([1, 1])}
Class: {'labels': array([0])}
Prediction: {'rosen': array([4.17030142])}
Local model predictions:
Local model 0: {'rosen': array([4.17030142])}
Local model 1: {'rosen': array([532.87631197])}
Local model 2: {'rosen': array([-46.32257302])}

Input value: {'x': array([[ 1,  1],
       [-2, -2]])}
Class: {'labels': array([[0],
       [2]])}
Prediction: {'rosen': array([[   4.17030143],
       [3608.99981798]])}
Local model predictions:
Local model 0: {'rosen': array([[  4.17030143],
       [409.40955704]])}
Local model 1: {'rosen': array([[ 532.87631197],
       [3514.58059766]])}
Local model 2: {'rosen': array([[ -46.32257302],
       [3608.99981798]])}

Plot clusters

Here, we plot the 10x10 = 100 Rosenbrock function data points, with colors representing the obtained clusters. The Rosenbrock function is represented by a contour plot in the background.

n_samples = dataset.n_samples
# Dataset is based on a DOE of 100=10^2 fullfact.
input_dim = int(sqrt(n_samples))
assert input_dim**2 == n_samples  # Check that n_samples is a square number

colors = ["b", "r", "g", "o", "y"]
inputs = dataset.get_data_by_group(dataset.INPUT_GROUP)
outputs = dataset.get_data_by_group(dataset.OUTPUT_GROUP)
x = inputs[:input_dim, 0]
y = inputs[:input_dim, 0]

Z = zeros((input_dim, input_dim))
for i in range(input_dim):
    Z[i, :] = outputs[input_dim * i : input_dim * (i + 1), 0]
fig = plt.figure()
cnt = plt.contour(x, y, Z, 50)
fig.colorbar(cnt)
for index in range(model.n_clusters):
    samples = nonzero(model.labels == index)[0]
    plt.scatter(inputs[samples, 0], inputs[samples, 1], color=colors[index])
plt.scatter(1, 1, marker="x")
plt.show()
plot moe

Plot data and predictions from final model

We construct a refined input space, and compute the model predictions.

refinement = 200

fine_x = linspace(x[0], x[-1], refinement)
fine_y = linspace(y[0], y[-1], refinement)
fine_x, fine_y = meshgrid(fine_x, fine_y)
fine_input = {"x": hstack([fine_x.flatten()[:, None], fine_y.flatten()[:, None]])}
fine_z = model.predict(fine_input)

# Reshape
fine_z = fine_z["rosen"].reshape((refinement, refinement))

plt.figure()
plt.imshow(Z)
plt.colorbar()
plt.title("Original data")
plt.show()

plt.figure()
plt.imshow(fine_z)
plt.colorbar()
plt.title("Predictions")
plt.show()
  • Original data
  • Predictions

Plot local models

for i in range(model.n_clusters):
    plt.figure()
    plt.imshow(
        model.predict_local_model(fine_input, i)["rosen"].reshape(
            (refinement, refinement)
        )
    )
    plt.colorbar()
    plt.title(f"Local model {i}")
    plt.show()
  • Local model 0
  • Local model 1
  • Local model 2

Total running time of the script: ( 0 minutes 3.414 seconds)

Gallery generated by Sphinx-Gallery