Notes on Baysian Optmization

Surrogate Models to the Rescue

If you have a cost function that is too expensive to evaluate, you should check out Bayesian Optimization.

The idea is to use a surrogate model to approximate the cost function and then use this model to find the best point to evaluate next.

The most common surrogate model is a Gaussian Process (GP), which is a distribution over functions. The GP is defined by its mean function $m(x)$ and covariance function $k(x,x^{\prime })$:

$$f(x)\sim \mathcal{GP}(m(x),k(x,x^{\prime }))$$

The GP is updated with the new data point and then used to find the next point to evaluate. This is typically done by maximizing an acquisition function, such as the Expected Improvement (EI):

$$EI(x)=\mathbb{E}[max(f(x)-f(x^{+}),0)]$$

where $f(x^{+})$ is the current best observed value.