Supervised learning

Supervised machine learning algorithm

Supervised machine learning is a task of learning relationships between input and output variables based on an input-output dataset. One usually distinguishes between to types of supervised machine learning algorithms, based on the nature of the outputs. For a continuous output variable, a regression is performed, while for a discrete output variable, a classification is performed.

Given a set of input variables \(x \in \mathbb{R}^{n_{\text{samples}}\times n_{\text{inputs}}}\) and a set of output variables \(y\in \mathbb{K}^{n_{\text{samples}}\times n_{\text{outputs}}}\), where \(n_{\text{inputs}}\) is the dimension of the input variable, \(n_{\text{outputs}}\) is the dimension of the output variable, \(n_{\text{samples}}\) is the number of training samples and \(\mathbb{K}\) is either \(\mathbb{R}\) or \(\mathbb{N}\) for regression and classification tasks respectively, a supervised learning algorithm seeks to find a function \(f: \mathbb{R}^{n_{\text{inputs}}} \to \mathbb{K}^{n_{\text{outputs}}}\) such that \(y=f(x)\).

In addition, we often want to impose some additional constraints on the function \(f\), mainly to ensure that it has a generalization capacity beyond the training data, i.e. it is able to correctly predict output values of new input values. This is called regularization. Assuming \(f\) is parametrized by a set of parameters \(\theta\), and denoting \(f_\theta\) the parametrized function, one typically seeks to minimize a function of the form

\[\mu(y, f_\theta(x)) + \Omega(\theta),\]

where \(\mu\) is a distance-like measure, typically a mean squared error or a cross entropy in the case of a regression, or a probability to be maximized in the case of a classification, and \(\Omega\) is a regularization term that limits the parameters from overfitting, typically some norm of its argument.

The supervised module implements this concept through the MLSupervisedAlgo class based on a Dataset.