Entropy

kl_div_mvn

pyvbmc.stats.kl_div_mvn(mu1, sigma1, mu2, sigma2)

Compute the analytical Kullback-Leibler divergence between two multivariate normal pdfs.

Parameters:

mu1np.ndarray

The k-dimensional mean vector of the first multivariate normal pdf.

sigma1np.ndarray

The covariance matrix of the first multivariate normal pdf.

mu2np.ndarray

The k-dimensional mean vector of the second multivariate normal pdf.

sigma2np.ndarray

The covariance matrix of the second multivariate normal pdf.

Returns:

kl_divnp.array

The forward and reverse Kullback-Leibler divergence. Both are infinite if either covariance matrix is singular or has a negative determinant, neither being the covariance of a distribution with a density.

entlb_vbmc

pyvbmc.entropy.entlb_vbmc(vp: VariationalPosterior, grad_flags: tuple = (True, True, True, True), jacobian_flag: bool = True)

Entropy lower bound for variational posterior by Jensen’s inequality.

Parameters:

vpVariationalPosterior

An instance of VariationalPosterior class.

grad_flagstuple of bool, len(grad_flags)=4, optional

Whether to compute the gradients for [mu, sigma, lambda, w].

jacobian_flagbool, optional

Whether variational parameters are transformed. The variational parameters and corresponding transformations are: sigma (log), lambda (log), w (softmax).

Returns:

H: float

Entropy lower bound of vp [1].

dH: np.ndarray

Gradient of entropy lower bound.

References

[1]

Gershman, S. J., Hoffman, M. D., & Blei, D. M. (2012). Nonparametric variational inference. Proceedings of the 29th International Conference on Machine Learning, 235–242.

entmc_vbmc

pyvbmc.entropy.entmc_vbmc(vp: VariationalPosterior, Ns: int, grad_flags: tuple = (True, True, True, True), jacobian_flag: bool = True, rng=None)

Monte Carlo estimate of entropy of variational posterior.

Parameters:

vpVariationalPosterior

An instance of VariationalPosterior class.

Nsint

Number of samples to draw per component. Ns > 0. Rounded up to an even number (antithetic sampling).

grad_flagstuple of bool, len(grad_flags)=4, default=tuple([True] * 4)

Whether to compute the gradients for [mu, sigma, lambda, w].

jacobian_flagbool

Whether variational parameters are transformed. The variational parameters and corresponding transformations are: sigma (log), lambda (log), w (softmax).

rngNone, int, SeedSequence or np.random.Generator, optional

Random generator (or seed) for the Monte Carlo samples. By default vp.rng is used.

Returns:

H: float

Estimated entropy of vp by Monte Carlo method.

dH: np.ndarray

Estimated entropy gradient by Monte Carlo method. \(dH = \left[\nabla_{\mu_1}^{T} H, ..., \nabla_{\mu_K}^{T} H, \nabla_{\sigma}^{T} H, \nabla_{\lambda}^{T} H, \nabla_{\omega}^{T} H\right]\) Only the blocks requested through grad_flags are present.

Notes

Ns antithetic samples are drawn from each component, the mixture density is evaluated at every sample, and H is the average of -log q weighted by the component weights; the gradients use the reparameterization trick (the sample of component j is mu_j + sigma_j * lambda * epsilon). All components’ samples come from one standard_normal((K, Ns / 2, D)) draw, which consumes the generator exactly as K successive draws of (Ns / 2, D) would. The density and the gradient terms are computed as a broadcast over a (components, samples, D, K) tensor of standardized distances, in blocks bounded by the fixed calibration profile (unless one sample’s D x K slab already exceeds it): blocks of components, and blocks of samples within a component when one component’s tensor alone exceeds the budget. The canonical blocks are the blocks of DEFAULT_CHUNK_ELEMENTS doubles, and the mixture densities and every sum over samples and components are taken over them whatever budget is in force, so neither the estimate nor its gradients depend on it: a computed block either subdivides one canonical block, whose component densities it fills in, or is a union of whole canonical blocks that it is split back into. The mixture sum over the components is a matrix-vector product, whose rounding follows the number of samples in a canonical block, so the estimate depends on the BLAS build at the level of its rounding.