PyBADS

PyBADS (Singh and Acerbi, 2024) is a Python implementation of the Bayesian Adaptive Direct Search (BADS) algorithm for solving difficult and moderately expensive optimization problems, previously implemented in MATLAB.

PyBADS is one of the open-source tools for fitting models to data from the Machine and Human Intelligence Group at the University of Helsinki. Check out our other tools, such as PyVBMC for the posterior and the model evidence and PyIBS for models that can only be simulated.

What is it?

BADS is a fast hybrid Bayesian optimization algorithm designed to solve difficult optimization problems, in particular related to fitting computational models (e.g., via maximum likelihood estimation).

BADS has been intensively tested for model fitting in science and engineering and is currently used in many computational labs around the world (see Google Scholar for many example applications).

In our benchmark with real model-fitting problems from computational and cognitive neuroscience, BADS performed on par or better than many other common and state-of-the-art optimizers, as shown in the original BADS paper (Acerbi and Ma, 2017).

BADS requires no specific tuning and runs off-the-shelf similarly to other Python optimizers, such as those in scipy.optimize.minimize.

Watch a three-minute film that explains how PyBADS works.

Note: If you are interested in estimating posterior distributions (i.e., uncertainty and error bars) over model parameters, and not just point estimates, you should check out Variational Bayesian Monte Carlo for Python (PyVBMC), a package for Bayesian posterior and model inference which can be used in synergy with PyBADS. Our model-fitting page explains when to use which.

What’s new in PyBADS 1.5

• Faster, with equal or better results. On our benchmark problems, PyBADS’s own computations, all that a run does besides evaluating the target, run almost twice as fast as in PyBADS 1.1.0; runs also need fewer evaluations, and find equal or better minima.

• Periodic variables. The option periodic_vars lists the indices of the variables that are periodic, such as angles; set the hard bounds of each one period apart, and BADS wraps the variable around them (see Example 6 and the FAQ).

• Fixed variables. A variable whose four bounds, hard and plausible, are equal is fixed at that value, and BADS optimizes the others (see the FAQ).

• Evaluations made before the run. BADS(..., precomputed_evaluations=(X, y)) gives a run the evaluations of the target that you already have, for instance from an earlier run: its Gaussian process uses them, and they do not count against max_fun_evals. The run still starts from x0 (see the BADS reference).

• Closer to MATLAB BADS. PyBADS was checked line by line against MATLAB BADS 1.1.3, the reference implementation, and follows it more closely in many details, above all with noisy targets. The FAQ lists what differs from MATLAB BADS.

• Requirements. PyBADS needs gpyreg 1.4.0 or later, NumPy 2.0 or later, SciPy 1.13 or later and matplotlib 3.9 or later.

• FAQ, tips and a coding-agent skill. The documentation has a page of frequently asked questions; a run occasionally prints a short tip with a link to the documentation (options={"show_tips": False} turns them off); and the PyBADS skill points a coding agent to the documentation relevant to its task: give the agent that file, or copy the skills/pybads folder into its skill directory.

The changelog lists what changed since PyBADS 1.1.0. Results differ from 1.1.0, also with a fixed seed. BADS checks the values of many options when it is created and refuses some that 1.1.0 accepted, and some calls and returned fields change: the changelog’s list “Upgrading from 1.1.0” says what to check in an existing script.

How does it work?

Tip

Watch a three-minute film that explains how PyBADS works.

PyBADS/BADS follows a mesh adaptive direct search (MADS) procedure for function minimization that alternates poll steps and search steps (see Fig 1).

• In the poll stage, points are evaluated on a mesh by taking steps in one direction at a time, until an improvement is found or all directions have been tried. The step size is doubled in case of success, halved otherwise.

• In the search stage, a Gaussian process (GP) is fit to a (local) subset of the points evaluated so far. Then, we iteratively choose points to evaluate according to a lower confidence bound strategy that trades off between exploration of uncertain regions (high GP uncertainty) and exploitation of promising solutions (low GP mean).

Fig 1: BADS procedure. The poll’s steps along each variable scale with that variable’s plausible range.

See here for a visualization of several optimizers at work, including BADS.

See our paper for more details (Acerbi and Ma, 2017).

Should I use PyBADS?

BADS is particularly recommended for problems in which:

• the objective function landscape is rough (nonsmooth), typically due to numerical approximations or noise;

• the objective function is at least moderately expensive to compute (e.g., more than 0.1 s per function evaluation);

• the gradient is unavailable;

• the number of input parameters is up to about D = 20.

The FAQ says what to use for other problems.

How-to

• Installation
• Getting started
• PyBADS: Frequently Asked Questions
• Examples
  • PyBADS Example 1: Basic usage
  • PyBADS Example 2: Non-box constraints
  • PyBADS Example 3: Noisy objective function
  • PyBADS Example 4: Noisy objective with user-provided noise estimates
  • PyBADS Example 5: Extended usage
  • PyBADS Example 6: Periodic variables
• Documentation
  • BADS
  • OptimizeResult
  • BADS options
  • Advanced documentation

Contributing

• Instructions for developers and contributors

References

1. Singh, G. S. & Acerbi, L. (2024). PyBADS: Fast and robust black-box optimization in Python. Journal of Open Source Software, 9(94), 5694. (paper on JOSS).

2. Acerbi, L. & Ma, W. J. (2017). Practical Bayesian Optimization for Model Fitting with Bayesian Adaptive Direct Search. In Advances in Neural Information Processing Systems 30: 1834-1844. (paper + supplement on arXiv, NeurIPS Proceedings)

Please cite both references if you use PyBADS in your work (the 2017 paper introduced the framework, and the latest one is its Python library). You can cite PyBADS in your work with something along the lines of

We optimized the log likelihoods of our models using Bayesian adaptive direct search (BADS; Acerbi and Ma, 2017), via the PyBADS software (Singh and Acerbi, 2024). PyBADS alternates between a series of fast, local Bayesian optimization steps and a systematic, slower exploration of a mesh grid.

BibTeX

@article{singh2024pybads,
  title={{PyBADS}: {F}ast and robust black-box optimization in {P}ython},
  author={Gurjeet Sangra Singh and Luigi Acerbi},
  publisher = {The Open Journal},
  journal = {Journal of Open Source Software},
  year = {2024},
  volume = {9},
  number = {94},
  pages = {5694},
  url = {https://doi.org/10.21105/joss.05694},
  doi = {10.21105/joss.05694}
}

@article{acerbi2017practical,
  title={Practical {B}ayesian Optimization for Model Fitting with {B}ayesian Adaptive Direct Search},
  author={Acerbi, Luigi and Ma, Wei Ji},
  journal={Advances in Neural Information Processing Systems},
  volume={30},
  pages={1834--1844},
  year={2017}
}

License and source

PyBADS is released under the terms of the BSD 3-Clause License. The Python source code is on GitHub. You may also want to check out the original MATLAB toolbox, and the lab’s other methods.

Acknowledgments:

PyBADS is developed by members (past and current) of the Machine and Human Intelligence Group at the University of Helsinki and ELLIS Institute Finland. Work on the PyBADS package is supported by the Research Council of Finland (grants 356498 and 358980 to Luigi Acerbi) and its Flagship programme: Finnish Center for Artificial Intelligence FCAI.

Starting from version 1.1, development of PyBADS has been assisted by coding agents, including Anthropic’s Claude Opus 5.5.