Experiments

The purpose of this module is to compare more than one models. Furthermore, this module can also optimize the hyper-parameters of these models and compare them. The Experiments class provides the basic building block for conducting experiments. The MLRegressionExperiments and MLClassificationExperiments compare several classical machine learning regression and classification models respectively. The DLRegressionExperiments class compares some common basic deep learning algorithms for a given data.

Experiments

class ai4water.experiments.Experiments(cases: Optional[dict] = None, exp_name: Optional[str] = None, num_samples: int = 5, verbosity: int = 1, monitor: Optional[Union[str, list]] = None)[source]

Bases: object

Base class for all the experiments.

All the expriments must be subclasses of this class. The core idea of Experiments is based upon model. An experiment consists of one or more models. The models differ from each other in their structure/idea/concept/configuration. When ai4water.experiments.Experiments.fit() is called, each model is built and trained. The user can customize, building and training process by subclassing this class and customizing ai4water.experiments.Experiments._build() and ai4water.experiments.Experiments._fit() methods.

- metrics
- exp_path
- model_
- models
- fit
- taylor_plot
- loss_comparison
- plot_convergence
- from_config
- compare_errors
- plot_improvement
- compare_convergence
- plot_cv_scores
- fit_with_tpot
__init__(cases: Optional[dict] = None, exp_name: Optional[str] = None, num_samples: int = 5, verbosity: int = 1, monitor: Optional[Union[str, list]] = None)[source]
Parameters
  • cases – python dictionary defining different cases/scenarios

  • exp_name – name of experiment, used to define path in which results are saved

  • num_samples – only relevent when you wan to optimize hyperparameters of models using grid method

  • verbosity (bool, optional) – determines the amount of information

  • monitor (str, list, optional) – list of performance metrics to monitor. It can be any performance metric SeqMetrics_ <https://seqmetrics.readthedocs.io> library. By default r2, corr_coeff, ``mse, rmse, r2_score, nse, kge, mape, pbias, bias mae, nrmse mase are considered for regression and accuracy, precision recall are considered for classification.

compare_convergence(show: bool = True, save: bool = False, name: str = 'convergence_comparison', **kwargs) Optional[matplotlib.axes._axes.Axes][source]

Plots and compares the convergence plots of hyperparameter optimization runs. Only valid if run_type=optimize during ai4water.experiments.Experiments.fit() call.

Parameters
  • show – whether to show the plot or now

  • save – whether to save the plot or not

  • name – name of file to save the plot

  • kwargs – keyword arguments to plot function

Returns

  • if the optimized models are >1 then it returns the maplotlib axes

  • on which the figure is drawn otherwise it returns None.

Examples

>>> from ai4water.experiments import MLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> experiment = MLRegressionExperiments()
>>> experiment.fit(data=busan_beach(), run_type="optimize", num_iterations=30)
>>> experiment.compare_convergence()
compare_errors(matric_name: str, cutoff_val: Optional[float] = None, cutoff_type: Optional[str] = None, save: bool = True, sort_by: str = 'test', ignore_nans: bool = True, name: str = 'ErrorComparison', show: bool = True, **kwargs) pandas.core.frame.DataFrame[source]

Plots a specific performance matric for all the models which were run during [fit][ai4water.experiments.Experiments.fit] call.

Parameters
  • matric_name – performance matric whose value to plot for all the models

  • cutoff_val – if provided, only those models will be plotted for whome the matric is greater/smaller than this value. This works in conjuction with cutoff_type.

  • cutoff_type – one of greater, greater_equal, less or less_equal. Criteria to determine cutoff_val. For example if we want to show only those models whose $R^2$ is > 0.5, it will be ‘max’.

  • save – whether to save the plot or not

  • sort_by – either test or train. How to sort the results for plotting. If ‘test’, then test performance matrics will be sorted otherwise train performance matrics will be sorted.

  • ignore_nans – default True, if True, then performance matrics with nans are ignored otherwise nans/empty bars will be shown to depict which models have resulted in nans for the given performance matric.

  • name – name of the saved file.

  • show (whether to show the plot at the end or not?) –

  • kwargs

    • fig_height :

    • fig_width :

    • title_fs :

    • xlabel_fs :

    • color :

Returns

pandas dataframe whose index is models and has two columns with name ‘train’ and ‘test’ These columns contain performance metrics for the models..

Return type

pd.DataFrame

Example

>>> from ai4water.experiments import MLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> data = busan_beach()
>>> inputs = list(data.columns)[0:-1]
>>> outputs = list(data.columns)[-1]
>>> experiment = MLRegressionExperiments(input_features=inputs, output_features=outputs)
>>> experiment.fit(data=data)
>>> experiment.compare_errors('mse')
>>> experiment.compare_errors('r2', 0.2, 'greater')
eval_best(data, model_type, opt_dir, **kwargs)[source]

Evaluate the best models.

fit(data, run_type: str = 'dry_run', opt_method: str = 'bayes', num_iterations: int = 12, include: Union[None, list] = None, exclude: Union[None, list, str] = '', cross_validate: bool = False, post_optimize: str = 'eval_best', hpo_kws: Optional[dict] = None)[source]

Runs the fit loop for all the models of experiment. The user can however, specify the models by making use of include and exclud keywords.

todo, post_optimize not working for ‘eval_best’ with ML methods.

Parameters
  • data – this will be passed to ai4water.Model.fit().

  • run_type – One of dry_run or optimize. If dry_run, the all the models will be trained only once. if optimize, then hyperparameters of all the models will be optimized.

  • opt_method – which optimization method to use. options are bayes, random, grid. Only valid if run_type is optimize

  • num_iterations – number of iterations for optimization. Only valid if run_type is optimize.

  • include – name of models to included. If None, all the models found will be trained and or optimized.

  • exclude – name of models to be excluded

  • cross_validate – whether to cross validate the model or not. This depends upon cross_validator agrument to the Model.

  • post_optimize – one of eval_best or train_best. If eval_best, the weights from the best models will be uploaded again and the model will be evaluated on train, test and all the data. If train_best, then a new model will be built and trained using the parameters of the best model.

  • hpo_kws – keyword arguments for ai4water.hyperopt.HyperOpt class.

Examples

>>> from ai4water.experiments import MLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> exp = MLRegressionExperiments()
>>> exp.fit(data=busan_beach())

If you want to compare only RandomForest, XGBRegressor, CatBoostRegressor and LGBMRegressor, use the include keyword

>>> exp.fit(data=busan_beach(), include=['RandomForestRegressor', 'XGBRegressor',
>>>    'CatBoostRegressor', 'LGBMRegressor'])

Similarly, if you want to exclude certain models from comparison, you can use exclude keyword

>>> exp.fit(data=busan_beach(), exclude=["SGDRegressor"])

if you want to perform cross validation for each model, we must give the cross_validator argument which will be passed to ai4water Model

>>> exp = MLRegressionExperiments(cross_validator={"KFold": {"n_splits": 10}})
>>> exp.fit(data=busan_beach(), cross_validate=True)

Setting cross_validate to True will populate cv_scores_ dictionary which can be accessed as exp.cv_scores_

if you want to optimize the hyperparameters of each model,

>>> exp.fit(data=busan_beach(), run_type="optimize", num_iterations=20)
fit_with_tpot(data, models: Optional[Union[int, List[str], dict, str]] = None, selection_criteria: str = 'mse', scoring: Optional[str] = None, **tpot_args)[source]

Fits the tpot’s fit method which finds out the best pipline for the given data.

Parameters
  • data

  • models

    It can be of three types.

    • If list, it will be the names of machine learning models/

      algorithms to consider.

    • If integer, it will be the number of top

      algorithms to consider for tpot. In such a case, you must have first run .fit method before running this method. If you run the tpot using all available models, it will take hours to days for medium sized data (consisting of few thousand examples). However, if you run first .fit and see for example what are the top 5 models, then you can set this argument to 5. In such a case, tpot will search pipeline using only the top 5 algorithms/models that have been found using .fit method.

    • if dictionary, then the keys should be the names of algorithms/models

      and values shoudl be the parameters for each model/algorithm to be optimized.

    • You can also set it to all consider all models available in

      ai4water’s Experiment module.

    • default is None, which means, the tpot_config argument will be None

  • selection_criteria – The name of performance metric. If models is integer, then according to this performance metric the models will be choosen. By default the models will be selected based upon their mse values on test data.

  • scoring (the performance metric to use for finding the pipeline.) –

  • tpot_args – any keyword argument for tpot’s Regressor or Classifier class. This can include arguments like generations, population_size etc.

Return type

the tpot object

Example

>>> from ai4water.experiments import MLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> exp = MLRegressionExperiments(exp_name=f"tpot_reg_{dateandtime_now()}")
>>> exp.fit(data=busan_beach())
>>> tpot_regr = exp.fit_with_tpot(busan_beach(), 2, generations=1, population_size=2)
classmethod from_config(config_path: str, **kwargs) ai4water.experiments._main.Experiments[source]

Loads the experiment from the config file.

Parameters
  • config_path – complete path of experiment

  • kwargs – keyword arguments to experiment

Returns

an instance of Experiments class

loss_comparison(loss_name: str = 'loss', include: Optional[list] = None, save: bool = True, show: bool = True, figsize: Optional[int] = None, start: int = 0, end: Optional[int] = None, **kwargs) matplotlib.axes._axes.Axes[source]

Plots the loss curves of the evaluated models. This method is only available if the models which are being compared are deep leanring mdoels.

Parameters
  • loss_name (str, optional) – the name of loss value, must be recorded during training

  • include – name of models to include

  • save – whether to save the plot or not

  • show – whether to show the plot or now

  • figsize (tuple) – size of the figure

  • **kwargs

    any other keyword arguments to be passed to the plot

Return type

matplotlib axes

Example

>>> from ai4water.experiments import DLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> data = busan_beach()
>>> exp = DLRegressionExperiments(
>>> input_features = data.columns.tolist()[0:-1],
>>> output_features = data.columns.tolist()[-1:],
>>> epochs=300,
>>> train_fraction=1.0,
>>> y_transformation="log",
>>> x_transformation="minmax",
>>> )
>>> exp.fit(data=data)
>>> exp.loss_comparison()

you may wish to plot on log scale

>>> exp.loss_comparison(logy=True)
plot_cv_scores(show: bool = False, name: str = 'cv_scores', exclude: Optional[Union[str, list]] = None, include: Optional[Union[str, list]] = None, **kwargs) Optional[matplotlib.axes._axes.Axes][source]

Plots the box whisker plots of the cross validation scores.

This plot is only available if cross_validation was set to True during ai4water.experiments.Experiments.fit().

Parameters
  • show (whether to show the plot or not) –

  • name (name of the plot) –

  • include (models to include) –

  • exclude (models to exclude) –

  • kwargs (any of the following keyword arguments) –

    • notch

    • vert

    • figsize

    • bbox_inches

Return type

matplotlib axes if the figure is drawn otherwise None

Example

>>> from ai4water.experiments import MLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> exp = MLRegressionExperiments(cross_validator={"KFold": {"n_splits": 10}})
>>> exp.fit(data=busan_beach(), cross_validate=True)
>>> exp.plot_cv_scores()
plot_improvement(metric_name: str, plot_type: str = 'dumbell', save: bool = True, name: str = '', dpi: int = 200, **kwargs) pandas.core.frame.DataFrame[source]

Shows how much improvement was observed after hyperparameter optimization. This plot is only available if run_type was set to optimize in ai4water.experiments.Experiments.fit().

Parameters
  • metric_name – the peformance metric for comparison

  • plot_type (str, optional) – the kind of plot to draw. Either dumbell or bar

  • save (bool) – whether to save the plot or not

  • name (str, optional) –

  • dpi (int, optional) –

  • **kwargs

    any additional keyword arguments for dumbell plot

Return type

pd.DataFrame

Examples

>>> from ai4water.experiments import MLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> experiment = MLRegressionExperiments()
>>> experiment.fit(data=busan_beach(), run_type="optimize", num_iterations=30)
>>> experiment.plot_improvement('r2')

or draw dumbell plot

>>> experiment.plot_improvement('r2', plot_type='bar')
sort_models_by_metric(metric_name, cutoff_val=None, cutoff_type=None, ignore_nans: bool = True, sort_by='test') pandas.core.frame.DataFrame[source]

returns the models sorted according to their performance

taylor_plot(include: Union[None, list] = None, exclude: Union[None, list] = None, figsize: tuple = (9, 7), **kwargs) matplotlib.figure.Figure[source]

Compares the models using taylor_plot.

Parameters
  • include (str, list, optional) – if not None, must be a list of models which will be included. None will result in plotting all the models.

  • exclude (str, list, optional) – if not None, must be a list of models which will excluded. None will result in no exclusion

  • figsize (tuple, optional) –

  • **kwargs – all the keyword arguments for taylor_plot function.

Return type

plt.Figure

Example

>>> from ai4water.experiments import MLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> data = busan_beach()
>>> inputs = list(data.columns)[0:-1]
>>> outputs = list(data.columns)[-1]
>>> experiment = MLRegressionExperiments(input_features=inputs, output_features=outputs)
>>> experiment.fit(data=data)
>>> experiment.taylor_plot()
train_best(data, model_type)[source]

Train the best model.

RegressionExperiments

class ai4water.experiments.MLRegressionExperiments(param_space=None, x0=None, cases=None, exp_name='MLRegressionExperiments', num_samples=5, verbosity=1, **model_kwargs)[source]

Bases: ai4water.experiments._main.Experiments

Compares peformance of 40+ machine learning models for a regression problem. The experiment consists of models which are run using fit() method. A model is one experiment.

The user can define new models by subclassing this class. In fact any new method in the sub-class which starts with model_ wll be considered as a new model. Otherwise the user has to overwite the attribute models to redefine, which methods (of class) are to be used as models and which should not. The method which is a model must only return key word arguments which will be streamed to the Model using build_and_run method. Inside this new method the user must define, which parameters to optimize, their param_space for optimization and the initial values to use for optimization.

__init__(param_space=None, x0=None, cases=None, exp_name='MLRegressionExperiments', num_samples=5, verbosity=1, **model_kwargs)[source]

Initializes the class

Parameters
  • param_space – dimensions of parameters which are to be optimized. These can be overwritten in models.

  • list (x0) – initial values of the parameters which are to be optimized. These can be overwritten in models

  • str (exp_name) – name of experiment, all results will be saved within this folder

  • dict (model_kwargs) – keyword arguments which are to be passed to Model and are not optimized.

Examples

>>> from ai4water.datasets import busan_beach
>>> from ai4water.experiments import MLRegressionExperiments
>>> # first compare the performance of all available models without optimizing their parameters
>>> data = busan_beach()  # read data file, in this case load the default data
>>> inputs = list(data.columns)[0:-1]  # define input and output columns in data
>>> outputs = list(data.columns)[-1]
>>> comparisons = MLRegressionExperiments(
...       input_features=inputs, output_features=outputs,
...       nan_filler= {'method': 'KNNImputer', 'features': inputs} )
>>> comparisons.fit(data=data,run_type="dry_run")
>>> comparisons.compare_errors('r2')
>>> # find out the models which resulted in r2> 0.5
>>> best_models = comparisons.compare_errors('r2', cutoff_type='greater',
...                                                cutoff_val=0.3)
>>> best_models = [m[1] for m in best_models.values()]
>>> # now build a new experiment for best models and otpimize them
>>> comparisons = MLRegressionExperiments(
...     inputs_features=inputs, output_features=outputs,
...     nan_filler= {'method': 'KNNImputer', 'features': inputs},
...     exp_name="BestMLModels")
>>> comparisons.fit(data=data, run_type="optimize", include=best_models)
>>> comparisons.compare_errors('r2')
>>> comparisons.taylor_plot()  # see help(comparisons.taylor_plot()) to tweak the taylor plot
class ai4water.experiments.DLRegressionExperiments(input_features: list, param_space=None, x0=None, cases: Optional[dict] = None, exp_name: Optional[str] = None, num_samples: int = 5, verbosity: int = 1, **model_kws)[source]

Bases: ai4water.experiments._main.Experiments

a framework for comparing several basic DL architectures for a given data.

To check the available models >>> exp = DLRegressionExperiments(…) >>> exp.models

If learning rate, batch size, and lookback are are to be optimzied, their space can be specified in the following way: >>> exp = DLRegressionExperiments(…) >>> exp.lookback_space = [Integer(1, 100, name=’lookback’)]

Example

>>> from ai4water.experiments import DLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> data = busan_beach()
>>> exp = DLRegressionExperiments(
>>> input_features = data.columns.tolist()[0:-1],
>>> output_features = data.columns.tolist()[-1:],
>>> epochs=300,
>>> train_fraction=1.0,
>>> y_transformation="log",
>>> x_transformation="minmax",
>>> )
>>> exp.fit(data=data)
__init__(input_features: list, param_space=None, x0=None, cases: Optional[dict] = None, exp_name: Optional[str] = None, num_samples: int = 5, verbosity: int = 1, **model_kws)[source]

initializes the experiment.

model_CNN(**kwargs)[source]

1D CNN based model

model_CNNLSTM(**kwargs)[source]

CNN-LSTM model

model_LSTM(**kwargs)[source]

LSTM based model

model_LSTMAutoEncoder(**kwargs)[source]

LSTM based auto-encoder model.

model_MLP(**kwargs)[source]

multi-layer perceptron model

model_TCN(**kwargs)[source]

Temporal Convolution network based model.

model_TFT(**kwargs)[source]

temporal fusion transformer model.

ClassificationExperiments

class ai4water.experiments.MLClassificationExperiments(param_space=None, x0=None, cases=None, exp_name='MLClassificationExperiments', num_samples=5, **model_kwargs)[source]

Bases: ai4water.experiments._main.Experiments

Runs classification models for comparison, with or without optimization of hyperparameters. It compares around 30 classification algorithms from sklearn, xgboost, catboost and lightgbm.

Examples

>>> from ai4water.datasets import MtropicsLaos
>>> from ai4water.experiments import MLClassificationExperiments
>>> data = MtropicsLaos().make_classification(lookback_steps=2)
>>> inputs = data.columns.tolist()[0:-1]
>>> outputs = data.columns.tolist()[-1:]
>>> exp = MLClassificationExperiments(input_features=inputs,
>>>                                       output_features=outputs)
>>> exp.fit(data=data, include=["CatBoostClassifier", "LGBMClassifier",
>>>             'RandomForestClassifier', 'XGBClassifier'])
>>> exp.compare_errors('accuracy', show=False)
__init__(param_space=None, x0=None, cases=None, exp_name='MLClassificationExperiments', num_samples=5, **model_kwargs)[source]
Parameters
  • param_space (list, optional) –

  • x0 (list, optional) –

  • cases (dict, optional) –

  • exp_name (str, optional) – name of experiment

  • num_samples (int, optional) –

  • **model_kwargs – keyword arguments for ai4water.Model class

DLRegressionExperiments

class ai4water.experiments.DLRegressionExperiments(input_features: list, param_space=None, x0=None, cases: Optional[dict] = None, exp_name: Optional[str] = None, num_samples: int = 5, verbosity: int = 1, **model_kws)[source]

Bases: ai4water.experiments._main.Experiments

a framework for comparing several basic DL architectures for a given data.

To check the available models >>> exp = DLRegressionExperiments(…) >>> exp.models

If learning rate, batch size, and lookback are are to be optimzied, their space can be specified in the following way: >>> exp = DLRegressionExperiments(…) >>> exp.lookback_space = [Integer(1, 100, name=’lookback’)]

Example

>>> from ai4water.experiments import DLRegressionExperiments
>>> from ai4water.datasets import busan_beach
>>> data = busan_beach()
>>> exp = DLRegressionExperiments(
>>> input_features = data.columns.tolist()[0:-1],
>>> output_features = data.columns.tolist()[-1:],
>>> epochs=300,
>>> train_fraction=1.0,
>>> y_transformation="log",
>>> x_transformation="minmax",
>>> )
>>> exp.fit(data=data)
model_CNN(**kwargs)[source]

1D CNN based model

model_CNNLSTM(**kwargs)[source]

CNN-LSTM model

model_LSTM(**kwargs)[source]

LSTM based model

model_LSTMAutoEncoder(**kwargs)[source]

LSTM based auto-encoder model.

model_MLP(**kwargs)[source]

multi-layer perceptron model

model_TCN(**kwargs)[source]

Temporal Convolution network based model.

model_TFT(**kwargs)[source]

temporal fusion transformer model.