BADS

Note

The BADS class implements the Bayesian Adaptive Direct Search (BADS) algorithm.

BADS attempts to solve an unbounded, bounded or nonlinearly constrained optimization (minimization) problem, and is compatible with both noiseless and noisy target functions.

To perform the optimization, first initialize a BADS object and then call bads.optimize() on the instance.

See below for more details on the BADS class methods and interface. The primary entry-points for users are the BADS class, which initializes the algorithm, and the OptimizeResult class, which represents the returned optimization solution. The Basic options may also be useful.

class pybads.bads.BADS(fun: callable, x0: ndarray = None, lower_bounds: ndarray = None, upper_bounds: ndarray = None, plausible_lower_bounds: ndarray = None, plausible_upper_bounds: ndarray = None, non_box_cons: callable = None, options: dict = None, *, gamma_uncertain_interval: float = None, precomputed_evaluations: tuple = None)

BADS Constrained optimization using Bayesian Adaptive Direct Search.

BADS attempts to solve problems of the form:

\(\mathtt{argmin}_x f(x)\) subject to: lower_bounds \(<= x <=\) upper_bounds, and optionally \(C(x) <= 0\)

Initialize a BADS object to set up the optimization problem, then run optimize(). See the examples for more details under the examples directory.

Parameters:

funcallable

The target function to minimize. fun takes a point x, a one-dimensional array of all the variables (the fixed ones included), and returns its value as a finite real number; with options['specify_target_noise'], it returns a tuple (f, sd), the value and the SD of its noise at x. In case the target function fun requires additional data/parameters, they can be handled using an anonymous function. For example: fun_for_pybads = lambda x: fun(x, data, extra_params), where fun is the function to optimize, and data and extra_params are given in the outer scope.

x0np.ndarray, optional

Starting point for the optimization, a single point of D elements, of shape (D,) or (1, D). If not specified or None, or if an element is not finite (nan, inf or -inf) at a variable that is not fixed (see below), the starting point x0 is drawn at random, over the variables that are not fixed, inside the plausible box between plausible_lower_bounds and plausible_upper_bounds (see below). With non_box_cons, a point that violates the constraints is drawn again, up to 1000 draws in all.

lower_bounds, upper_boundsnp.ndarray, optional

lower_bounds (lb) and upper_bounds (ub) define a set of strict lower and upper bounds for the coordinate vector, x, so that the unknown function has support on lb <= x <= ub. If scalars, the bound is replicated in each dimension. Use None for lb and ub if no bounds exist. Set lb[i] = -inf if the i-th coordinate is unbounded below, and ub[i] = inf if it is unbounded above (while other coordinates may be bounded). Note that if lb and/or ub contain infinite bounds, the respective values of plb and/or pub need to be specified (see below). By default None.

plausible_lower_bounds, plausible_upper_boundsnp.ndarray, optional

Specifies a set of plausible_lower_bounds (plb) and plausible_upper_bounds (pub) such that lb <= plb < pub <= ub at each variable that is not fixed (see below). Both plb and pub need to be finite, and are replicated in each dimension if scalars. If not specified, plb is lb and pub is ub: plb needs to be specified when lb has an infinite bound, and pub when ub has one. plb and pub represent a plausible range, which should denote a region where the global minimum is expected to be found. As a rule of thumb, set plausible_lower_bounds and plausible_upper_bounds such that there is > 90% probability that the minimum is found within the box (where in doubt, just set plb = lb and pub = ub).

A variable whose four bounds are equal is fixed at their value; x0 holds that value there, or a non-finite value. BADS optimizes the other variables, and the defaults of the options that depend on the number of variables count only those. The points you pass to BADS or receive from it hold all the variables, the fixed ones at their values: those that fun, non_box_cons and options['output_fcn'] receive, the result’s x and x0, the points "x" of iteration_history and the points of precomputed_evaluations; the indices of options['periodic_vars'] count all the variables.

non_box_conscallable, optional

A given non-box constraints function that specifies constraint violations. It takes an array of shape (N, D), one point per row in the original space, and returns an array of shape (N,) or (N, 1), one value per point, true or positive where the point violates the constraints. For example, lambda x: np.sum(x**2, axis=1) > 1 keeps the search inside the unit ball. A feasible region thinner than the mesh can resolve, such as a band narrower than the poll steps, can leave every point of the initial design and of the polls infeasible, and the run can then end early, near x0, on the stall criterion (an improvement below tol_fun over tol_stall_iters iterations). Reparametrize such a problem so that its feasible region is wide: for the band abs(x[0] - x[1]) <= w, for example, optimize over (x[0] + x[1]) / 2 and (x[0] - x[1]) / w, the latter bounded by -1 and 1 in place of the constraint.

optionsdict, optional

Additional options can be passed as a dict. Please refer to the BADS options page for the default options. If no options are passed, the default options are used. To run BADS on a noisy (stochastic) objective function, set options['uncertainty_handling'] = True. You can help BADS by providing an estimate of the noise. options['noise_size'] = sigma provides a global estimate of the SD of the noise in your problem in a good region of the parameter space. (If not specified, default sigma = 1.0). Alternatively, you can specify the target noise at each location with options['specify_target_noise'] = True. In this case, fun is expected to return two values, the estimate of the target at x and an estimate of the SD of the noise at x (see the examples). If options['uncertainty_handling'] is not specified, BADS will determine at runtime if the objective function is noisy, or turn uncertainty handling on with options['specify_target_noise'] = True. A variable that is periodic, such as an angle, is named in options['periodic_vars'], a list of the indices of the variables, from 0, the fixed ones included: its hard bounds, which need to be finite, are its period, and BADS wraps it around them, so that lb and ub are the same point and the variable is optimized across them. To obtain reproducible results of the optimization, set options['random_seed'] to a fixed integer (see rng below).

gamma_uncertain_intervalfloat, optional, keyword-only

With options['stobads'], the multiplier of the half-width of the uncertainty interval of the Sto-BADS success rule. By default None, which is 1.96. Deprecated along with options['stobads'], which is experimental and may be removed in a future release.

precomputed_evaluationstuple, optional, keyword-only

Evaluations of fun made before the run, for instance by an earlier run, as a tuple (or a list) (X, y), or (X, y, y_sd) with options['specify_target_noise'], which requires y_sd. X holds one point per row in the original space, of shape (N, D), each within the hard bounds and satisfying non_box_cons; y holds the values of fun at them and y_sd the SDs of their noise, both of shape (N,). The values are finite and the SDs positive. BADS uses them as training data of its Gaussian process, and does not evaluate again the points of its initial design that they hold; they do not count toward max_fun_evals (func_count). The run still starts from x0: its first incumbent is the best of the points it evaluates itself (pass the best of the evaluations as x0 to start there). Unless options['uncertainty_handling'] is True (or options['specify_target_noise'] is), a point given twice must have the same value, and is kept once; otherwise each repeat is an observation of its own. By default None, no evaluations.

Attributes:

rngnumpy.random.Generator

The generator of every random draw of the run, including the random x0. It is created with the BADS object from options['random_seed'], which takes what numpy.random.default_rng takes, such as a non-negative integer or a SeedSequence, or a Generator, used as given. If the option is None (default), the generator is derived from NumPy’s global random state, so that np.random.seed before creating the BADS object fixes the run. The run does not otherwise draw from NumPy’s global random state: a target that draws from it is not fixed by random_seed. On Apple Silicon Macs, two runs with the same seed can end at slightly different points.

Raises:

ValueError

When x0 is not specified and neither plausible_lower_bounds nor lower_bounds is, or neither plausible_upper_bounds nor upper_bounds is: a missing plausible bound defaults to the hard bound, and the random x0 is drawn between the plausible bounds.

ValueError

When x0 or the bounds fail the checks of their shapes, values and order: for instance an x0 of more than one row, plausible bounds that are not finite, bounds out of the order lb <= plb < pub <= ub at a variable that is not fixed, a finite x0 that differs from the value of a fixed variable, or bounds that fix every variable.

ValueError

When non_box_cons, given an (N, D) array, does not return an array of shape (N,) or (N, 1), or when x0 violates the constraints, or each of the 1000 random draws of it does.

ValueError

When an option has an unknown name or a value that BADS does not take: for instance a max_fun_evals, max_iter or tol_stall_iters that is neither a positive integer nor inf, a tol_mesh that is not a positive finite number, a noise_size that is not one or two numbers, a value other than True or False for uncertainty_handling or for an option whose default is one of them (plot excepted), a periodic_vars that is not a list of distinct indices of variables with finite bounds, or an f_vals that holds a finite value, a non-empty fun_values, or acq_hedge=True, options that are not supported.

ValueError

When precomputed_evaluations is not a tuple (or a list) of two or three arrays of the shapes above, of finite values and positive SDs, when it has y_sd without options['specify_target_noise'] or lacks them with it, when one of its points lies outside the hard bounds or violates non_box_cons, or when a point given twice has two different values where a point is kept once.

ValueError

When options['random_seed'] is a negative integer.

TypeError

When options['random_seed'] is a float that is not a whole number, a string, or another value that numpy.random.default_rng does not take.

References

[1]

Singh, S. G. & Acerbi, L. (2024). “PyBADS: Fast and robust black-box optimization in Python”. Journal of Open Source Software, 9(94), 5694, https://doi.org/10.21105/joss.05694.

[2]

Acerbi, L. & Ma, W. J. (2017). “Practical Bayesian Optimization for Model Fitting with Bayesian Adaptive Direct Search”. In Advances in Neural Information Processing Systems 30, pages 1834-1844. (arXiv preprint: https://arxiv.org/abs/1705.04405).

Examples

For BADS usage examples, please look up the Jupyter notebook tutorials in the PyBADS documentation: https://acerbilab.github.io/pybads/examples.html

optimize()

Run the optimization on an initialized BADS object.

BADS starts at x0 and finds a local minimum x of the target function fun.

A history of the optimization problem can be found in the iteration_history attribute of the BADS object.

A BADS object runs a single optimization: create a new one for each run.

Returns:

optimize_resultOptimizeResult

Dictionary containing the result of the optimization. See the documentation of the OptimizeResult class for more details. For example, retrieve the final solution and its value with the attributes optimize_result.x and optimize_result.fval.

Raises:

RuntimeError

If optimize has already been called on this object, whether or not that run completed.

ValueError

If fun returns a value that is not a finite real number, or, with options['specify_target_noise'], does not return a tuple (f, sd) with a finite positive sd. An error that fun raises reaches the caller.