HyperParameter Optimization

This module is for optimization of hyper-parameters. The HyperOpt class performs optimization by minimizing the objective which is defined by a user defined objective function. The space of hyperparameters can be defined by using Categorical, Integer and Real classes.

For tutorial on using this class, see this hpo_tutorial notebook

Categorical

class ai4water.hyperopt.Categorical(categories, prior=None, transform=None, name=None)

This class is used when parameter has distinct group/class of values such as [1,2,3] or [‘a’, ‘b’, ‘c’]. This class overrides skopt’s Categorical class. It Can be converted to optuna’s distribution or hyper_opt’s choice. It uses same input arguments as received by skopt’s Categorical class

- as_hp

- to_optuna

- suggest

- to_optuna

- serialize

Example

>>> from ai4water.hyperopt import Categorical
>>> activations = Categorical(categories=['relu', 'tanh', 'sigmoid'], name='activations')

__init__(categories, prior=None, transform=None, name=None)

serialize()

Serializes the Categorical object so that it can be saved in json

Real

class ai4water.hyperopt.Real(low: Optional[float] = None, high: Optional[float] = None, num_samples: Optional[int] = None, step: Optional[int] = None, grid: Optional[Union[list, ndarray]] = None, *args, **kwargs)

This class is used for the parameters which have fractional values such as real values from 1.0 to 3.5. This class extends the Real class of Skopt so that it has an attribute grid which then can be fed to optimization algorithm to create grid space. It also adds several further methods to it.

grid

- as_hp

- to_optuna

- suggest

- to_optuna

- serialize

Example

>>> from ai4water.hyperopt import Real
>>> lr = Real(low=0.0005, high=0.01, prior='log', name='lr')

__init__(low: Optional[float] = None, high: Optional[float] = None, num_samples: Optional[int] = None, step: Optional[int] = None, grid: Optional[Union[list, ndarray]] = None, *args, **kwargs)

Parameters

• low – lower limit of parameter

• high – upper limit of parameter

• step – used to define grid in conjuction with low and high

• grid – array like, if given, low, high, step and num_samples will be redundant.

• num_samples – if given, it will be used to create grid space using the formula

serialize()

Serializes the Real object so that it can be saved in json

to_optuna()

returns an equivalent optuna space

Integer

class ai4water.hyperopt.Integer(low: int = None, high: int = None, num_samples: int = None, step: int = None, grid: np.ndarray, list = None, *args, **kwargs)

This class is used when the parameter is integer such as integer values from 1 to 10. Extends the Real class of Skopt so that it has an attribute grid which then can be fed to optimization algorithm to create grid space. Moreover it also generates optuna and hyperopt compatible/equivalent instances.

grid

- as_hp

- to_optuna

- suggest

- to_optuna

- serialize

Example:

>>> from ai4water.hyperopt import Integer
>>> units = Integer(low=16, high=128, name='units')

__init__(low: int = None, high: int = None, num_samples: int = None, step: int = None, grid: np.ndarray, list = None, *args, **kwargs)

Parameters

• low – lower limit of parameter

• high – upper limit of parameter

• list/array (grid) – If given, low and high should not be given as they will be calculated from this grid.

• int (num_samples) – if given , it will be used to calculated grid using the formula np.arange(low, high, step)

• int – if given, it will be used to create grid space using the formula np.linspace(low, high, num_samples)

serialize()

Serializes the Integer object so that it can be saved in json

to_optuna()

returns an equivalent optuna space

HyperOpt

class ai4water.hyperopt.HyperOpt(algorithm: str, *, param_space, objective_fn, eval_on_best: bool = False, backend: Optional[str] = None, opt_path: Optional[str] = None, process_results: bool = True, verbosity: int = 1, **kwargs)

Bases: object

The purpose of this class is to provide a uniform and simplifed interface to use hyperopt, optuna, scikit-optimize and scikit-learn based hyperparameter optimization methods. Ideally this class should provide all the functionalities of beforementioned libaries with a uniform interface. It however also complements these libraries by combining their functionalities and adding some additional functionalities to them. On the other hand this class should not limit or complicate the use of its underlying libraries. This means all the functionalities of underlying libraries are available in this class as well. Moreover, you can use this class just as you use one of its underlying library.

The purpose here is to make a class which allows application of any of the available optimization methods on any type of model/classifier/regressor. If the classifier/regressor is of sklearn-based, then for random search, we use RanddomSearchCV, for grid search, we use GridSearchCV and for Bayesian, we use BayesSearchCV . On the other hand, if the model is not sklearn-based, you will still be able to implement any of the three methods. In such case, the bayesian will be implemented using gp_minimize. Random search and grid search will be done by simple iterating over the sample space generated as in sklearn based samplers. However, the post-processing of the results is (supposed to be) done same as is done in RandomSearchCV and GridSearchCV.

The class is expected to pass all the tests written in sklearn or skopt for corresponding classes.

For detailed use of this class see this hpo_tutorial

- results dict

- gpmin_results dict

- skopt_results

- hp_space

- space

- skopt_space

- space dict

- title str

default this is same as name of algorithm. For AI4Water based

models, this is more detailed, containing problem type etc.

Type

name of the folder in which all results will be saved. By

- eval_with_best: evaluates the objective_fn on best parameters

- best_paras(): returns the best parameters from optimization.

The following examples illustrate how we can uniformly apply different optimization algorithms.

Examples

>>> from ai4water import Model
>>> from ai4water.hyperopt import HyperOpt, Categorical, Integer, Real
>>> from ai4water.datasets import busan_beach
>>> from SeqMetrics import RegressionMetrics
>>> data = busan_beach()
>>> input_features = ['tide_cm', 'wat_temp_c', 'sal_psu', 'air_temp_c', 'pcp_mm', 'pcp3_mm']
>>> output_features = ['tetx_coppml']

We have to define an objective function which will take keyword arguments and return a scaler value as output. This scaler value will be minized during optimzation

>>> def objective_fn(**suggestion)->float:
...   # the objective function must receive new parameters as keyword arguments
...    model = Model(
...        input_features=input_features,
...        output_features=output_features,
...        model={"XGBRegressor": suggestion},
...        train_data='random',
...        verbosity=0)
...
...    model.fit(data=data)
...
...    t, p = model.predict(return_true=True)
...    mse = RegressionMetrics(t, p).mse()
...    # the objective function must return a scaler value which needs to be minimized
...    return mse

Define search space The search splace determines pool from which parameters are chosen during optimization.

>>> num_samples=5   # only relavent for random and grid search
>>> search_space = [
...    Categorical(['gbtree', 'dart'], name='booster'),
...    Integer(low=1000, high=2000, name='n_estimators', num_samples=num_samples),
...    Real(low=1.0e-5, high=0.1, name='learning_rate', num_samples=num_samples)
...]
...
... # Using Baysian with gaussian processes
>>> optimizer = HyperOpt('bayes', objective_fn=objective_fn, param_space=search_space,
...                     num_iterations=num_iterations )
>>> optimizer.fit()

Using TPE with optuna

>>> num_iterations = 10
>>> optimizer = HyperOpt('tpe', objective_fn=objective_fn, param_space=search_space,
...                     backend='optuna',
...                     num_iterations=num_iterations )
>>> optimizer.fit()

Using cmaes with optuna

>>> optimizer = HyperOpt('cmaes', objective_fn=objective_fn, param_space=search_space,
...                     backend='optuna',
...                     num_iterations=num_iterations )
>>> optimizer.fit()

Using random with optuna, we can also try hyperopt and sklearn as backend for random algorithm

>>> optimizer = HyperOpt('random', objective_fn=objective_fn, param_space=search_space,
...                     backend='optuna',
...                     num_iterations=num_iterations )
>>> optimizer.fit()

Using TPE of hyperopt

>>> optimizer = HyperOpt('tpe', objective_fn=objective_fn, param_space=search_space,
...                     backend='hyperopt',
...                     num_iterations=num_iterations )
>>> optimizer.fit()

Using grid with sklearn

>>> optimizer = HyperOpt('grid', objective_fn=objective_fn, param_space=search_space,
...                     backend='sklearn',
...                     num_iterations=num_iterations )
>>> optimizer.fit()

Backward compatability The following shows some tweaks with hyperopt to make its working compatible with its underlying libraries. using grid search with AI4Water

using Bayesian with custom objective_fn

>>> def f(x, noise_level=0.1):
...      return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) + np.random.randn() * noise_level
...
>>> opt = HyperOpt("bayes",
...           objective_fn=f,
...           param_space=[Categorical([32, 64, 128, 256], name='lstm_units'),
...                        Categorical(categories=["relu", "elu", "leakyrelu"], name="dense_actfn")
...                        ],
...           acq_func='EI',  # Expected Improvement.
...           n_calls=50,     #number of iterations
...           x0=[32, "relu"],  # inital value of optimizing parameters
...           n_random_starts=3,  # the number of random initialization points
...           )
>>> opt_results = opt.fit()

using Bayesian with custom objective_fn and named args

>>> def f(noise_level=0.1, **kwargs):
...    x = kwargs['x']
...    return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) + np.random.randn() * noise_level
>>> opt = HyperOpt("bayes",
...           objective_fn=f,
...           param_space=[Categorical([32, 64, 128, 256], name='lstm_units'),
...                        Categorical(categories=["relu", "elu", "leakyrelu"], name="dense_actfn")
...                        ],
...           acq_func='EI',  # Expected Improvement.
...           n_calls=50,     #number of iterations
...           x0=[32, "relu"],  # inital value of optimizing parameters
...           n_random_starts=3,  # the number of random initialization points
...           random_state=2
...           )
>>> opt_results = opt.fit()

__init__(algorithm: str, *, param_space, objective_fn, eval_on_best: bool = False, backend: Optional[str] = None, opt_path: Optional[str] = None, process_results: bool = True, verbosity: int = 1, **kwargs)

Initializes the class

Parameters

• algorithm (str) – must be one of random, grid, bayes, bayes_rf, and tpe, defining which optimization algorithm to use.

• objective_fn (callable) – Any callable function whose returned value is to be minimized. It can also be either sklearn/xgboost based regressor/classifier.

• param_space (list, dict) – the search space of parameters to be optimized. We recommend the use of Real, Integer and categorical classes from [ai4water.hyperopt][ai4water.hyperopt.Integer] (not from skopt.space). These classes allow a uniform way of defining the parameter space for all the underlying libraries. However, to make this class work exactly similar to its underlying libraries, the user can also define parameter space as is defined in its underlying libraries. For example, for hyperopt based method like ‘tpe’ the parameter space can be specified as in the examples of hyperopt library. In case the code breaks, please report.

• eval_on_best (bool, optional) – if True, then after optimization, the objective_fn will be evaluated on best parameters and the results will be stored in the folder named “best” inside title folder.

• opt_path – path to save the results

• backend (str, optional) – Defines which backend library to use for the algorithm. For example the user can specify whether to use optuna or hyper_opt or sklearn for grid algorithm.

• verbosity (bool, optional) – determines amount of information being printed

• **kwargs – Any additional keyword arguments will for the underlying optimization algorithm. In case of using AI4Water model, these must be arguments which are passed to AI4Water’s Model class.

__getattr__(item)

add_previous_results(iterations: Optional[Union[dict, str]] = None, x: Optional[list] = None, y: Optional[list] = None)

adds results from previous iterations.

If you have run the optimization priviously, you can make use of those results by appending them.

Parameters

• iterations – It can be either a dictionary whose keys are y values and values are x or it can be a path to a file which contains these xy values as dictioary.

• x – a list of lists where each sub-list is the value of hyperparameter at at one iteratio. The x and y arguments optional and will only be used if iterations are not provided.

• y – a list of float values where each value in y is the output of objective_fn with corresponding x. The length of x and y must be equal.

property backend

best_iter() → int

returns the iteration on which best/optimized parameters are obtained. The indexing starts from 0.

best_paras(as_list=False) → Union[list, dict]

best_xy() → dict

Returns best (optimized) parameters as dictionary. The dictionary has two keys x and y. x is the best hyperparameters while y is the corresponding objective function value.

check_args(**kwargs)

static dict_to_xy(iterations: dict)

dims()

eval_sequence(params)

eval_with_best()

Find the best parameters and evaluate the objective_fn with them. :param return_model bool: If True, then then the built objective_fn will be returned

fit(*args, **kwargs)

Makes and calls the underlying fit method

fmin(**kwargs)

classmethod from_gp_parameters(fpath: str, objective_fn)

loads results saved from bayesian optimization

func_vals() → ndarray

returns the value of objective function at each iteration.

grid_search()

hp_space() → dict

returns a dictionary whose values are hyperopt equivalent space instances.

model_for_gpmin()

This function can be called in two cases

• The user has made its own objective_fn.

• We make objective_fn using AI4Water and return the error.

In first case, we just return what user has provided.

property num_iterations

property objective_fn_is_dl

property opt_path

optuna_objective(**kwargs)

optuna_study() → Study

Attempts to create an optuna Study instance so that optuna based plots can be generated. Returns None, if not possible.

original_para_order()

own_fit()

property param_space

plot_importance(raise_error=True, save=True, plot_type='box')

pre_calculated_results(resutls, from_gp_parameters=True)

Loads the pre-calculated results i.e. x and y values which have been already evaluated.

process_results()

post processing of results

random_search()

property random_state

save_iterations_as_xy()

serialize()

skopt_results()

skopt_space()

Tries to make skopt compatible Space object. If unsuccessful, return None

space() → dict

Returns a skopt compatible space but as dictionary

property title

to_kw(x)

property use_named_args

property use_own

property use_sklearn

property use_skopt_bayes

property use_skopt_gpmin

property use_tpe

xy_of_iterations() → Dict[int, Dict[str, Union[str, dict]]]

returns a dictionary whose keys are iteration numbers are values are xy parirs at those iterations.

Returns Dict[int, Dict[str, [dict,float]]]