PyBADS Example 4: Noisy objective with user-provided noise estimates#
In this example, we will show how to run PyBADS on a noisy target for which we can estimate the noise at each evaluation.
This notebook is Part 4 of a series of notebooks in which we present various example usages for BADS with the PyBADS package. The code used in this example is available as a script here.
import sys
if "google.colab" in sys.modules: # Colab lacks PyBADS: install it
%pip install "pybads>=1.5.1"
import numpy as np
from pybads import BADS
1. Problem setup#
We assume you are already familiar with optimization of noisy targets with PyBADS, as described in the previous notebook.
Sometimes, you may be able to estimate the noise associated with each function evaluation, for example via bootstrap or more sophisticated estimation methods such as inverse binomial sampling (PyIBS, one of the lab’s tools for fitting models to data). If you can do that, it is highly recommended you do so and tell PyBADS by activating the specify_target_noise option.
In the user-specified noise case, the target function fun is expected to return two outputs:
the (noisy) estimate of the function at
x(as usual);the estimate of the standard deviation of the noisy evaluation at
x.
In this toy example below, we know the standard deviation sigma by construction. Note that the function is heteroskedastic, that is, the noise depends on the input location x.
As in the previous notebook, the example is reproducible: random_seed fixes the random seed of the run, and the target draws its noise from its own random number generator, noise_rng, which we create with a fixed seed right before the run, since random_seed does not seed the target (the PyBADS FAQ says how to make a run reproducible).
def noisy_sphere_estimated_noise(x, scale=1.0):
"""Quadratic function with heteroskedastic noise; also return noise estimate."""
x_2d = np.atleast_2d(x)
f = np.sum(x_2d**2, axis=1)
sigma = scale * (1.0 + np.sqrt(f))
y = f + sigma * noise_rng.normal(size=x_2d.shape[0])
return y, sigma
x0 = np.array([-3, -3]) # Starting point
lower_bounds = np.array([-5, -5])
upper_bounds = np.array([5, 5])
plausible_lower_bounds = np.array([-2, -2])
plausible_upper_bounds = np.array([2, 2])
options = {
"uncertainty_handling": True,
"specify_target_noise": True,
"noise_final_samples": 100,
"random_seed": 0, # Makes the run reproducible
}
2. Run the optimization#
As usual, we run bads with the user-defined options, after creating noise_rng; running this cell again repeats the run.
noise_rng = np.random.default_rng(1) # Makes the target's noise reproducible
bads = BADS(
noisy_sphere_estimated_noise,
x0,
lower_bounds,
upper_bounds,
plausible_lower_bounds,
plausible_upper_bounds,
options=options,
)
optimize_result = bads.optimize()
Beginning optimization of a STOCHASTIC objective function (specified noise)
Tip: To hold a parameter at a known value, set its four bounds (lb, ub, plb and pub) to that value, and its entry of x0 to the same value or NaN. PyBADS optimizes only the other parameters and still passes all of them to your objective, so your code needs no change.
https://acerbilab.github.io/pybads/faq.html#faq-can-i-set-lb-ub-for-some-variable-to-fix-it-to-a-given-value
Iteration f-count E[f(x)] SD[f(x)] MeshScale Method Actions
0 1 19.8118 5.24264 1
0 33 -4.2962 5.24264 1 Initial mesh Initial points
1 37 -4.2962 2.85814 0.5 Refine grid Train
2 45 -4.2962 2.85814 0.25 Refine grid Train
3 46 -0.369928 0.584932 0.25 Successful search (ES-ell)
3 57 -0.369928 0.584932 0.125 Refine grid Train
4 58 0.107542 0.39513 0.125 Successful search (ES-ell)
4 59 -0.0270465 0.385564 0.125 Successful search (ES-ell)
4 63 -0.0655998 0.360467 0.125 Incremental search (ES-ell)
4 64 -0.155686 0.354577 0.125 Successful search (ES-ell)
4 66 -0.180293 0.308933 0.125 Incremental search (ES-ell)
4 73 -0.180293 0.308933 0.0625 Refine grid
5 81 -0.178771 0.222092 0.03125 Refine grid
6 83 -0.171116 0.226618 0.03125 Successful search (ES-wcm)
6 93 -0.171116 0.226618 0.015625 Refine grid
7 94 -0.105285 0.182654 0.015625 Successful search (ES-ell)
7 96 -0.12971 0.184857 0.015625 Successful search (ES-ell)
7 99 -0.143946 0.191024 0.015625 Successful search (ES-ell)
7 109 -0.143946 0.191024 0.0078125 Refine grid
8 111 -0.0990663 0.16602 0.0078125 Successful search (ES-ell)
8 121 -0.0990663 0.16602 0.00390625 Refine grid
9 122 -0.0764383 0.146798 0.00390625 Successful search (ES-ell)
9 123 -0.116683 0.148587 0.00390625 Successful search (ES-ell)
9 133 -0.116683 0.148587 0.00195312 Refine grid
10 136 -0.104299 0.133233 0.00195312 Successful search (ES-wcm)
10 137 -0.105469 0.133381 0.00195312 Successful search (ES-wcm)
10 138 -0.115483 0.134088 0.00195312 Successful search (ES-wcm)
10 140 -0.126765 0.133586 0.00195312 Successful search (ES-wcm)
10 141 -0.13744 0.135497 0.00195312 Successful search (ES-wcm)
10 149 -0.13744 0.135497 0.000976562 Refine grid
11 150 -0.121907 0.121191 0.000976562 Successful search (ES-wcm)
11 153 -0.127559 0.119214 0.000976562 Successful search (ES-wcm)
11 154 -0.136879 0.118329 0.000976562 Successful search (ES-wcm)
11 155 -0.140811 0.117566 0.000976562 Successful search (ES-wcm)
11 159 -0.152085 0.115159 0.000976562 Successful search (ES-wcm)
11 169 -0.152085 0.115159 0.000488281 Refine grid
12 170 -0.104212 0.109687 0.000488281 Incremental search (ES-wcm)
12 171 -0.104519 0.109168 0.000488281 Incremental search (ES-wcm)
12 175 -0.11347 0.107256 0.000976562 Successful poll Train
13 178 -0.118734 0.106033 0.000976562 Successful search (ES-ell)
13 187 -0.118734 0.106033 0.000488281 Refine grid Train
14 189 -0.0886045 0.101715 0.000488281 Successful search (ES-wcm)
14 191 -0.0894262 0.10102 0.000488281 Incremental search (ES-wcm)
14 194 -0.0911504 0.0999564 0.000488281 Successful search (ES-wcm)
14 196 -0.102475 0.0993563 0.000488281 Successful search (ES-wcm)
14 207 -0.102475 0.0993563 0.000244141 Refine grid Train
15 209 -0.0868465 0.13522 0.000244141 Successful search (ES-wcm)
15 210 -0.0901129 0.134696 0.000244141 Successful search (ES-ell)
15 213 -0.0919201 0.131797 0.000244141 Successful search (ES-ell)
15 214 -0.108964 0.135302 0.000244141 Successful search (ES-ell)
15 215 -0.110016 0.135103 0.000244141 Successful search (ES-ell)
15 216 -0.128623 0.134553 0.000244141 Successful search (ES-ell)
15 217 -0.133759 0.134378 0.000244141 Successful search (ES-ell)
15 218 -0.135862 0.134561 0.000244141 Successful search (ES-ell)
15 219 -0.139821 0.135453 0.000244141 Successful search (ES-ell)
15 227 -0.160174 0.133862 0.000488281 Successful poll Train
16 229 -0.171962 0.132654 0.000488281 Successful search (ES-ell)
16 230 -0.181168 0.132647 0.000488281 Successful search (ES-ell)
16 234 -0.197749 0.12471 0.000488281 Successful search (ES-ell)
16 237 -0.198251 0.123128 0.000488281 Incremental search (ES-ell)
16 239 -0.202391 0.123165 0.000488281 Successful search (ES-ell)
16 240 -0.216061 0.123465 0.000488281 Successful search (ES-ell)
16 241 -0.216078 0.124104 0.000488281 Incremental search (ES-ell)
16 242 -0.228517 0.126788 0.000488281 Successful search (ES-ell)
16 247 -0.228947 0.121658 0.000488281 Incremental search (ES-ell)
16 251 -0.228947 0.121658 0.000244141 Refine grid
17 254 -0.220688 0.10938 0.000244141 Successful search (ES-ell)
17 263 -0.223984 0.104399 0.000488281 Successful poll
18 271 -0.246568 0.104737 0.000244141 Refine grid
19 279 -0.255861 0.10072 0.00012207 Refine grid
20 287 -0.265891 0.0982914 6.10352e-05 Refine grid Train
21 295 -0.217379 0.0964613 3.05176e-05 Refine grid Train
22 303 -0.222965 0.0937663 1.52588e-05 Refine grid
23 311 -0.210283 0.103449 7.62939e-06 Refine grid
24 319 -0.171727 0.0891701 3.8147e-06 Refine grid
25 327 -0.188306 0.086611 1.90735e-06 Refine grid Train
26 334 -0.21005 0.0851576 9.53674e-07 Refine grid Train
Optimization terminated: mesh size less than options['tol_mesh'].
Estimated function value at minimum: 0.06217438017824448 ± 0.1034447582854115 (precision-weighted mean ± SEM from 100 samples)
3. Results and conclusions#
As per noisy target optimization, optimize_result['fval'] is the estimated function value at optimize_result['x'], here obtained by taking the weighted mean of the final sampled evaluations (each evaluation is weighted by its precision, or inverse variance).
The final samples can be found in optimize_result['yval_vec'], and their estimated standard deviation in optimize_result['ysd_vec'].
x_min = optimize_result["x"]
fval = optimize_result["fval"]
fsd = optimize_result["fsd"]
print(
f"BADS minimum at: x_min = {x_min.flatten()}, fval (estimated) = {fval:.4g} +/- {fsd:.2g}"
)
print(
f"total f-count: {optimize_result['func_count']}, time: {round(optimize_result['total_time'], 2)} s"
)
print(f"final evaluations (shape): {optimize_result['yval_vec'].shape}")
print(f"final evaluations SD (shape): {optimize_result['ysd_vec'].shape}")
BADS minimum at: x_min = [-0.00393684 0.03422188], fval (estimated) = 0.06217 +/- 0.1
total f-count: 434, time: 6.56 s
final evaluations (shape): (100,)
final evaluations SD (shape): (100,)
We can also check the ground-truth value of the target function at the returned point once we remove the noise:
print(
f"The true, noiseless value of f(x_min) is {noisy_sphere_estimated_noise(x_min,scale=0)[0][0]:.3g}."
)
The true, noiseless value of f(x_min) is 0.00119.
Compare this to the true global minimum of the sphere function at \(\textbf{x}^\star = [0,0]\), where \(f^\star = 0\).
Remarks#
Due to the elevated level of noise, we do not necessarily expect high precision in the solution.
For more information on optimizing noisy objective functions, see the PyBADS FAQ.