CMA-ES: My favorite black-box optimizer

CMA-ES, invented by Nikolaus Hansen and Andreas Ostermeier and refined over three decades, is a standard black-box optimizer for problems where gradients do not exist, suffer from severe noise, or cost hours per evaluation, such as aircraft CFD, suspension bridge FEA, or discrete neural architecture search. Instead of stepping a single point downhill, CMA-ES maintains a Gaussian search distribution, shifting its mean toward better samples and bending its covariance matrix toward a multiple of the objective's inverse Hessian, discovering landscape curvature without calculating derivatives.

CMA-ES: My favorite black-box optimizer When gradients are cheap and smooth, Adam and SGD excel. But when gradients do not exist, suffer from severe noise, or cost hours per evaluation , such as in aircraft CFD, suspension bridge FEA, or discrete neural architecture search, CMA-ES is the standard workhorse. Instead of stepping a single point downhill, CMA-ES maintains a Gaussian search distribution , shifting its mean toward better samples and bending its covariance matrix toward a multiple of the objective's inverse Hessian, discovering landscape curvature without calculating derivatives. Invented by Nikolaus Hansen and Andreas Ostermeier, and refined over three decades by Hansen and collaborators https://www.cmap.polytechnique.fr/~nikolaus.hansen/ g f x . Ax + b once the initial distribution is transformed to match. H⁻¹ C approaches a multiple of the inverse Hessian without derivatives. See the Math technical-addendum What CMA-ES is and why anyone should care In deep learning, optimization almost always means gradients : Adam, Adafactor, Lion, and SGD with cosine schedules evaluate backpropagation derivatives to step downhill. In engineering design, physical robotics, and scientific simulations, gradients often do not exist, suffer from severe discretization noise, or cost hours of compute per point . In black-box optimization, where you can only evaluate a system and observe its scalar output, CMA-ES Covariance Matrix Adaptation Evolution Strategy is the standard workhorse. CMA-ES provides a coordinate-invariant, sample-efficient method to navigate opaque, non-differentiable objective landscapes into a good basin; on strongly multimodal problems, restart strategies IPOP/BIPOP supply the global sweep. Interactive 2D Gaussian Distribution Sandbox Drag the distribution parameters manually, or run live step-by-step CMA-ES evolution down the valley Adapts via CSA path length. Stretches covariance along the ill-conditioned valley. Rotates leading eigenvector to align with valley curvature. As generations progress, the blue confidence ellipse elongates and rotates to match the curvature of the ravine this sandbox caps the drawn eigenvalue ratio at 8:1 for readability , while the green arrow pulls the mean along the valley floor without evaluating gradients. Live Simulation Internal Algebraic StateStatic State Real-time mathematical parameters governing the Gaussian distribution manifold The Core Mathematical Philosophy At its foundation, CMA-ES operates under a clean abstraction: “You do not optimize a single point in parameter space. You optimize aprobability distributionover parameter space, and iteratively reshape that distribution to concentrate probability mass where performance is highest.” Suppose you want to minimize an unknown black-box objective function . You supply CMA-ES with the dimension , an initial mean vector , and an initial step size . 1. The Gaussian Search Distribution Sampling Each candidate design is generated by taking the current distribution center , scaling the search radius by step size , rotating along principal axes , stretching along standard deviations , and perturbing by white Gaussian noise . Candidate Offspring SampleCandidate Offspring Sample / Point Dimension: n-dimensional vector The concrete parameter vector sent to your simulator or black-box evaluator. One scout deployed into parameter space. A batch of λ such vectors forms the current generation. After evaluation, designs are sorted strictly by relative performance rank. Instead of searching blindly, CMA-ES uses linear algebra B D to transform simple spherical white noise into an elongated ellipsoid that mirrors the curvature of the optimization landscape. In each generation , the algorithm executes a four-beat cycle: Sampling Offspring Generate a batch of λ candidate designs from the current multivariate normal distribution. Initially, C is spherical; over successive iterations, it stretches along the valley floor. Black-Box Evaluation Evaluate the simulator, finite-element solver, or hyperparameter training run for each candidate vector to obtain scalar performance scores. Rank-Based Selection Sort the population by relative rank. CMA-ES ignores raw magnitudes and tracks only order, providing invariance to any strictly increasing monotonic transformation g f x . Distribution Update Shift mean m toward weighted elite designs, adapt covariance matrix C via rank-1 trajectory memory and rank-μ batch spread, and adjust step size σ using path momentum. 2. Rank-Based Recombination & Weighted Mean Shift The next generation mean is computed as a weighted average of the top using logarithmically decreasing rank weights . Updated Distribution MeanDistribution Mean / Anchor Dimension: n-dimensional vector The new center of probability mass for generation g + 1. Shifts base camp smoothly into the basin carved by the best-performing candidates. Moves in the direction of the sampled natural gradient without calculating explicit derivatives. Because weights depend solely on index rank rather than raw score differences, applying any strictly increasing transformation g f x produces the exact same update. Outliers and scaling distortions cannot derail the search. Why CMA-ES Implicitly Learns Inverse-Hessian Geometry On an ill-conditioned quadratic bowl , isotropic search struggles because steep directions oscillate while shallow directions crawl. Newton's method resolves this by preconditioning gradients with , transforming elliptical contours into spherical circles where steepest descent points directly at the minimum. CMA-ES discovers this geometry without forming a Hessian matrix or computing derivatives . The weighted covariance of selected steps the rank-µ update , reinforced by the evolution path rank-1 , adapts until, approximately: 8. Approximate Inverse-Hessian Curvature Alignment On any convex quadratic bowl with Hessian matrix , the learned covariance matrix empirically becomes approximately proportional to the inverse Hessian for displacements without calculating derivatives. Adapted Covariance MatrixCovariance Matrix / Curvature Dimension: n × n matrix The learned distribution shape maintained by CMA-ES. Stretches along flat valley floors and compresses across steep ridges. Transforms elliptical contours into spherical isotropic contours in transformed space. Newton's method preconditions the gradient with H⁻¹ to jump directly to the minimum. CMA-ES gains the same geometric advantage for derivative-free optimization by adapting C toward a multiple of H⁻¹. Sampling from whitens the search landscape, so the linear convergence rate becomes essentially independent of the problem's conditioning, where standard genetic algorithms or random search stall. Why I Love CMA-ES The five architectural pillars that make it the ultimate black-box workhorse Principled Distribution-Space Updates Maintains an explicit multivariate normal search distribution. Its mean shift and rank-μ covariance update together form a Monte Carlo natural-gradient step on the Riemannian manifold of Gaussians with the Fisher information metric; the evolution paths layer cross-generation history on top. Invariance to Monotone Objective Warping Rank-based selection evaluates only relative ordering, so scaling, shifting, or applying any strictly increasing transformation such as logarithmic or exponential rewards leaves the optimization path identical. Converts Expensive Evals into Geometry Learning When each simulation run is expensive, every sample must count. CMA-ES reuses every ranked batch to learn curvature, adapting its covariance toward on quadratic bowls so it can navigate ill-conditioned ravines without finite-difference probing. Mixed Discrete/Continuous Versatility By normalizing parameters into a unit cube and decoding at simulation boundaries, CMA-ES handles continuous dimensions, quantized integers, and categorical choices in a single search vector. Integer and categorical coordinates need a lower bound on their step size or CMA-ES with margin so quantization plateaus cannot stall adaptation. Virtually Parameter-Free Out of the Box Default heuristics , CSA damping constants, recombination weights function reliably across diverse problem domains without tedious manual hyperparameter sweeps. When gradients disappear When people first encounter optimization without analytic gradients, a common suggestion is to approximate derivatives via finite differences. In production engineering pipelines, simulations contain turbulence switches, finite-element meshing boundaries, and categorical parameters. Finite difference quotients amplify localized noise into arbitrary vectors. Why Numerical Gradients Fail in Black Boxes Perturbation noise explodes finite differences f x+ε - f x / ε, while CMA-ES ranks whole neighborhoods Smaller explodes noise; larger causes severe truncation bias. With dimensions, forward finite differences require 21 simulation runs per step , and localized perturbation noise points the resulting vector in a misleading direction. CMA-ES estimates collective distribution shifts over the population to remain stable. 1. Transonic Aircraft Wing Aerodynamics 10–20 geometry parameters; hours per CFD run; discrete airfoil families - Navier-Stokes CFD meshing with turbulence boundary layer transitions - Blended scalar penalty: lift-to-drag ratio, wave drag, stall onset, root bending stress - Categorical NACA airfoil family indices introduce discrete step discontinuities - Grid search is intractable: 10 sample points across 15 dimensions requires 10¹⁵ evaluations 2. Nonlinear Structural FEA Suspension Bridge Mode crossings, wind vortex resonance, stress envelope constraints - Solves nonlinear elasticity systems under dead, live, wind, and seismic load combinations - Step changes occur when buckling modes cross or tensile stress exceeds the material's yield limit 250–1200 MPa depending on grade - Non-smooth constraint penalties degrade second-order Taylor approximations - Rank-based selection handles discontinuous penalty boundaries without numerical overflow 3. Neural Architecture & Mixed Hyperparameter Search Mixed continuous/discrete knobs, stochastic validation noise, budget ~200 evals - Continuous parameters: learning rate schedule, weight decay, SwiGLU beta, LayerNorm eps - Discrete architectural integers: layer count L ∈ 6..48 , d model, attention head counts - Evaluation requires multi-hour cluster training with stochastic validation scores - Unit-box encoding maps discrete choices into continuous search space without custom heuristics Interactive Case Study: Suspension Bridge Under Dynamic Load Live 3D FEA stress calculation with moving traffic & structural optimization Multi-Parameter Bridge Structural Optimization 8-Dimensional Mixed Continuous, Discrete & Categorical Parameter Search yσ y= 250 MPa p c Vector MemoryDiscrete topologies & materials are smoothly partitioned across continuous 0, 1 interval bins. Holographic Neural Architecture Search NAS 5D Mixed Discrete-Continuous CMA-ES minimizes a weighted loss-plus-compute objective in a continuous latent space; the chart tracks the empirical Pareto set of every architecture evaluated. Loss and latency come from hand-written surrogates, not training runs. GQA MLP: SwiGLU Params: 44.0M A concrete CMA-ES walk-through: designing an airplane wing Let's trace CMA-ES through a concrete engineering challenge: designing an optimal transonic aircraft wing across an 8-dimensional mixed parameter space spanning continuous planform geometry, discrete structural rib counts, and categorical airfoil family profiles. Each aerodynamic evaluation trades lift-to-drag against the structural mass driven by the wing root bending moment. The drag total inside already folds in transonic wave drag and skin friction. Transonic Wing Aerodynamic & Structural Multi-Parameter Optimization 8-Dimensional Mixed Continuous & Discrete Parameter Evolution p c Vector MemoryCMA-ES searches continuous geometry, sweep angles, and discrete airfoil families. CMA-ES Generation Lifecycle in Slow Motion Click through the 7 algorithmic phases or press Play to watch the live step-by-step cycle 1Initial State & Normalization Two high-performance CMA-ES engines in Rust High-performance production implementations in Rust, targeting WebAssembly for browser deployment and native C/Python bindings for cluster workloads. wasm cmaes WebAssembly + SIMD + Web Worker Parallelism Zero-overhead Rust CMA-ES compiled to WebAssembly. Features sequential and multi-threaded Rayon worker pools with SIMD matrix decomposition. - Two distribution bundles: pkg/ pure JS and pkg-par/ SharedArrayBuffer + Rayon . - Deterministic seeded LCG random number generator for reproducible simulation replays. fast cmaes Native Python Bindings + AVX-512 SIMD Acceleration A high-throughput Rust engine with a drop-in scipy.optimize compatible surface. 10–30× faster than pure Python implementations when the objective itself is cheap and interpreter overhead dominates. - Rich terminal UI displaying live step size σ , condition number κ C , and best fitness. - Full support for diagonal sep-CMA-ES, active covariance, boundary repair, and IPOP/BIPOP restarts. Live CMA-ES landscape explorer WASM Test CMA-ES live against classical benchmark test functions. Watch the covariance matrix adapt its principal axes, observe cumulative step-size adaptation σ grows on aligned steps and shrinks on oscillating ones , and compare its convergence rate against finite-difference gradient descent. A narrow, curved valley in which fixed-step first-order methods often zigzag across the walls while CMA-ES can rotate and elongate its search distribution along the valley. Live Simulation Internal Algebraic StatePaused / Stepping Real-time covariance eigensystem, evolution path momentum, and step-size adaptation Inside the optimizer: covariance geometry in 3D frankensim WASM Inside the optimizer: live covariance geometry Every generation of a real CMA-ES run, projected into 3D principal-component space. The ellipsoid is the 1σ surface of the sampling distribution N m, σ²C : watch it form and orient as the population learns the landscape. The view renormalizes overall size for visibility, so read σ's collapse from the telemetry. Discrete options are mapped to disjoint sub-ranges of a continuous 0, 1 variable — so one optimizer coordinate encodes a categorical choice, and the same machinery optimizes mixed continuous / integer / categorical design spaces. which direction the mean traveled ; the rank- term reshapes the ellipse around the elite cloud itself. Current landscape: Banana valley: CMA-ES must learn a curved, correlated ridge. Technical addendum: what is really going on This technical addendum covers the mathematical foundations of CMA-ES, synthesized from Nikolaus Hansen and Anne Auger's foundational publications, Akimoto et al.'s information-geometric natural gradient proofs, and practical production engineering considerations. The thesis of this addendum: CMA-ES is no ad-hoc biological heuristic. Its core update is a coordinate-invariant natural gradient step on the Riemannian manifold of multivariate Gaussian distributions. 1. The Search Distribution as an Optimization Object In classical optimization, one maintains a single candidate vector . In CMA-ES, one maintains a parameterized probability density on , where : 1. The Gaussian Search Distribution Sampling Each candidate design is generated by taking the current distribution center , scaling the search radius by step size , rotating along principal axes , stretching along standard deviations , and perturbing by white Gaussian noise . Candidate Offspring SampleCandidate Offspring Sample / Point Dimension: n-dimensional vector The concrete parameter vector sent to your simulator or black-box evaluator. One scout deployed into parameter space. A batch of λ such vectors forms the current generation. After evaluation, designs are sorted strictly by relative performance rank. Instead of searching blindly, CMA-ES uses linear algebra B D to transform simple spherical white noise into an elongated ellipsoid that mirrors the curvature of the optimization landscape. Why a Gaussian? By the Principle of Maximum Entropy, the multivariate normal distribution is the unique distribution that maximizes information entropy for a specified mean and covariance matrix. It represents the least committal prior under second-order ignorance. Covariance Metric Adaptation & Natural Gradient Alignment Comparing Euclidean steepest descent against the covariance-transformed Natural Gradient In ill-conditioned valleys, standard Euclidean descent red points perpendicular to the valley floor, causing catastrophic zig-zagging. The natural gradient mint , preconditioned by covariance , aims directly down the canyon. 2. Information Geometry & The Sampled Natural Gradient Suppose the objective is to maximize expected performance under the search distribution . Standard Euclidean steepest ascent on depends arbitrarily on how the distribution is parameterized such as Cholesky vs eigendecomposition vs matrix logarithm . To achieve coordinate invariance, information geometry equips the statistical manifold with the Fisher Information Metric : The canonical steepest ascent direction invariant to reparameterization is the Natural Gradient : 6. Natural Gradient on the Gaussian Statistical Manifold The canonical natural gradient on the probability distribution manifold transforms the raw Euclidean gradient through the inverse Fisher Information Matrix to achieve coordinate parameterization invariance. Natural Gradient DirectionCovariance Matrix / Curvature Dimension: Parameter space vector The steepest ascent direction of expected fitness J θ = E {x~P θ} -f x with respect to the Fisher information metric. The direction that maximizes expected fitness improvement for a fixed infinitesimal change in distribution Kullback-Leibler KL divergence. Akimoto et al. and the IGO framework of Ollivier et al. proved that the mean and rank-μ covariance updates are a sampled Monte Carlo natural gradient step; the evolution paths and CSA are refinements outside that derivation. CMA-ES is not an ad-hoc biological heuristic. Its core update the mean shift and the rank-μ covariance term is a natural gradient step on the Riemannian manifold of multivariate Gaussian distributions equipped with the Fisher Information metric; the evolution paths and step-size control add history that a single natural gradient step cannot capture. Akimoto et al. and Ollivier et al. demonstrated that when rank-based weights are substituted for raw fitness values , the CMA-ES mean update and rank-μ covariance update correspond exactly to a sampled natural gradient step on the Gaussian manifold. The evolution paths, the rank-1 term, and step-size adaptation accumulate cross-generation history that lies outside this derivation. 3. Cumulative Step-Size Adaptation CSA Adapting step size via standard 1/5th success rules fails in non-spherical landscapes. CMA-ES tracks an exponentially smoothed evolution path in whitened coordinate space: 3. Cumulative Step-Size Adaptation CSA Evolution Path The step-size evolution path accumulates consecutive mean shifts in whitened coordinate space , exponentially smoothed by factor with effective population mass . Conjugate Evolution PathEvolution Path / Momentum Memory Dimension: n-dimensional vector Exponentially weighted history vector tracking the directional alignment of consecutive generational steps. The momentum vector in isotropic coordinates. Measures whether the optimizer is running straight down a runway or zigzagging. Under neutral random selection, p σ is distributed as a standard Gaussian vector, so its expected length is E||N 0, I || ≈ √n. Why whiten by C^{-1/2}? Without whitening, an elongated covariance matrix would make steps along the major axis appear artificially long even when wandering. Whitening normalizes all directions into a perfect sphere. Under random selection in a neutral fitness landscape, behaves as a stationary Gaussian process with . The expected length of a standard normal vector serves as the baseline: 11. Expected Length of an n-Dimensional Standard Normal Vector The expected Euclidean length of pure random Gaussian noise in dimensions serves as the exact reference baseline for Cumulative Step-Size Adaptation. Expected Chi-Distribution LengthGaussian White Noise Source Dimension: Positive scalar ≈ √n The mean length of a sample from the standard normal distribution in n dimensions: E ||z|| . The natural radius of thermal brownian motion in an n-dimensional room. The CSA yardstick: path length above this baseline means aligned steps grow σ , near it means uncorrelated steps hold σ , below it means oscillating steps shrink σ . In high dimensions, a standard Gaussian's mass is not spread through a solid ball; it concentrates in a thin shell of roughly constant thickness around radius √n. CSA exploits this concentration to calibrate step sizes with precision. CSA compares the empirical path length to its expectation under random selection: 4. Exponential Step-Size Update Inertial Cruise Control The new step size multiplies the current step size by an exponential factor based on how much the path length exceeds the expected length of random noise , damped by constant . Updated Step SizeGlobal Step Size / Scale Dimension: Positive scalar The adapted global scale factor for generation g + 1. Grows exponentially when progress is consistently collinear; shrinks when trapped or near an optimum. Changes smoothly without sudden discrete step-halving jumps. If consecutive steps point in the same direction, ||p σ|| E||N 0, I ||, causing the exponent to be positive exp 0 1 , which expands σ. If steps oscillate or cancel, ||p σ|| < E||N 0, I ||, causing exp <0 < 1, which contracts σ. Consistently aligned steps: causes to increase accelerating across flat valleys . Oscillating or canceling steps: causes to decrease zooming in around local minima . 4. Covariance Matrix Adaptation Rank-1 & Rank-μ Updates The full covariance update blends historical memory, rank-1 momentum path , rank-μ batch spread, and active negative updates: 5. Covariance Matrix Adaptation Rank-1 + Rank-μ + Active CMA The updated covariance matrix blends historical memory , a rank-1 momentum update along the trajectory of the mean, a rank-μ update over the current elite cloud, and active negative weights to shrink harmful directions. Updated Covariance MatrixCovariance Matrix / Curvature Dimension: n × n matrix Symmetric positive-definite matrix encoding learned landscape conditioning. The geometric memory of the optimizer. On quadratic bowls it settles near C ∝ H⁻¹, up to stochastic fluctuations. Reshapes the spherical search cloud into an elongated ellipsoid aligned with low-cost ridges. CMA-ES combines two distinct timescales of learning: rank-1 updates exploit historical temporal correlations between generations, while rank-μ updates exploit spatial variance within the current generation. where the rank-1 anisotropic evolution path accumulates momentum in parameter coordinates: 7. Rank-1 Anisotropic Evolution Path Mean Trajectory Memory The anisotropic path accumulates consecutive mean steps in physical coordinates with decay , gated by Heaviside step indicator and population weight . Anisotropic Evolution PathEvolution Path / Momentum Memory Dimension: n-dimensional vector Accumulates directional correlations across generations in unwhitened parameter coordinates. The directional inertia of the distribution center. Drives rank-1 updates to stretch the covariance along long corridors. Unlike p σ, p c does not use whitening, preserving real physical orientation. By tracking mean movement across consecutive generations, p c detects consistent linear ridges that a single generation's offspring cloud could never resolve alone. Rank-1 Update : Exploits correlations between consecutive generations. It acts like an online Principal Component Analysis PCA along the trajectory of the mean. Rank- Update : Exploits intra-generation variance among the top elite points in the current batch; crucial for large parallel populations. Active CMA : Uses negative weights on the worst-ranked offspring ranks to shrink variance along harmful directions. 5. Fundamental Invariance Properties A central reason CMA-ES is so well-behaved under black-box assumptions is its dual invariance: 9. Strict Invariance to Monotone Objective Warping Applying any strictly increasing transformation with positive derivative to the objective preserves every selection ranking, so with the same random seed CMA-ES produces the identical sequence of means, step sizes, and covariance matrices. Monotone Warping FunctionGlobal Step Size / Scale Dimension: Scalar function R → R Any strictly increasing scalar mapping such as log f , exp f , or a piecewise scaling . Arbitrary distortion applied to objective scores by simulation metrics. Does not change the ordering of any pair of candidate designs. A nonlinear monotone rescaling such as exp f leaves the minimizer unchanged but can make gradients explode or vanish, so gradient methods must re-tune their step sizes. CMA-ES reads only the relative rank order, which the rescaling leaves untouched, so it takes the identical sequence of steps. 10. Affine Invariance Under Full Coordinate Rotations & Rescaling Transforming the search space coordinates from to by any invertible linear matrix and translation yields identical optimization trajectories in transformed coordinates, provided the initial mean and covariance are transformed the same way. Transformed Coordinate VectorDistribution Mean / Anchor Dimension: n-dimensional vector The search space under arbitrary rotation, shear, or scaling. How the problem appears in a different unit or coordinate frame. CMA-ES adapts C y = A C x Aᵀ to match the transformed geometry identically. Optimizers that treat each parameter independently coordinate descent, separable evolutionary algorithms, per-parameter step-size rules degrade badly if you rotate the coordinate system by 45 degrees, because parameters become cross-coupled. Because CMA-ES maintains the full covariance C with rotation matrix B, it treats all coordinate systems identically. 6. Multi-Dimensional Phase Space & Internals Lab Explore high-dimensional covariance adaptation and trajectory momentum in real time. Switch between canonical test landscapes, vary search dimensionality from 2D to 12D, and observe the live 3D PCA projection as CMA-ES whitens ill-conditioned ravines and settles into multimodal basins escaping them is the job of restart strategies . Inside the optimizer: live covariance geometry Every generation of a real CMA-ES run, projected into 3D principal-component space. The ellipsoid is the 1σ surface of the sampling distribution N m, σ²C : watch it form and orient as the population learns the landscape. The view renormalizes overall size for visibility, so read σ's collapse from the telemetry. Discrete options are mapped to disjoint sub-ranges of a continuous 0, 1 variable — so one optimizer coordinate encodes a categorical choice, and the same machinery optimizes mixed continuous / integer / categorical design spaces. which direction the mean traveled ; the rank- term reshapes the ellipse around the elite cloud itself. Current landscape: Banana valley: CMA-ES must learn a curved, correlated ridge. Interactive CMA-ES Hyperparameter & Budget Sizer Calculate exact Hansen default parameters, population sizes, and wall-clock budgets for your problem dimension n 10 parametersUnlike stochastic gradient descent which requires tuning learning rates, momentum, decay schedules, and weight decay for every model , CMA-ES computes all internal learning rates as deterministic functions of dimension and selection mass , so they are never tuned per problem. The choices left to you are the initial point , the initial step size about 0.3 times the parameter range , and optionally . - Work in an unconstrained space; logit/tanh to map back. If you clip/reflect at bounds instead, add a penalty on the repair distance so the boundary plateau cannot stall step-size adaptation. - Categories: carve 0,1 into intervals, quantize late; keeps search smoother. Unordered categories with many options often do better one-hot encoded. - Hard constraints: add rank-based penalties; repair samples instead of rejecting. - For noisy f: enlarge λ, reevaluate elites and average their fitness, or lower the two covariance learning rates. Active negative-weight updates amplify misranked samples, so they are not a noise remedy. - Budgeting: λ = 4 + ⌊3 ln n⌋; expect on the order of n to n² generations when C must adapt; restart if stalled. - Keep seeds and ask/tell logs so you can replay and debug; determinism saves days. Universal Encode/Decode Latent Box Mapping Mapping mixed continuous, log-scale, and discrete integer knobs into an isotropicunit cube By searching in the continuous unit box and only quantizing at the very moment of simulation evaluation, the probability distribution moves smoothly across discrete boundaries; the optimizer itself never sees the staircase. Two caveats apply. Once shrinks below a bin width, all offspring in that coordinate decode identically and selection goes blind, so integer coordinates need a step-size floor or CMA-ES with margin . And slicing one axis into bins imposes an ordering on the categories that may not exist, which is why unordered choices are often one-hot encoded instead. Noise Explorer & Stochastic Robustness How rank-based selection and population scaling filter out severe simulation noise Raising makes the noisy rank ordering more reliable, and the weighted recombination averages independent noise so the error in the mean update shrinks roughly like . Gradient-based algorithms rely on numerical difference ratios, which divide by near-zero step intervals and blow up in the presence of noise. CMA-ES only needs the relative rank ordering of samples, so the scale of the objective is irrelevant. Noise still corrupts the ordering itself, which is exactly why larger populations, elite reevaluation, and explicit uncertainty handling exist; try the Cauchy setting to watch heavy-tailed spikes scramble the ranks. Box Constraint Handling & Boundary Repair Navigating hard boundary constraints without destroying covariance conditioning When samples overshoot bounds , reflect them back into the interior . Preserves step variance without collapsing covariance eigenvalues. Production implementations pair any repair with a penalty on the repair distance and feed the unrepaired sample back to the update. Piles up multiple samples onto the boundary line . Wastes degrees of freedom and causes severe covariance matrix condition number degradation along normal vectors. Search operates in unbounded ; the smooth sigmoid maps onto the open box , so overshoot is impossible by construction. The trade-off: the map is asymptotic, so an optimum sitting exactly on the boundary like the corner target here is only approached as and progress flattens near the edges. Active Covariance Adaptation Negative Weights Actively pruning unpromising variance directions vs passive covariance discounting Standard CMA-ES expands covariance along directions that produce elite samples, but relies solely on passive exponential discounting to shrink variance in bad directions. Active CMA-ES assigns negative weights to the worst-ranked offspring, with each contribution rescaled by its Mahalanobis length so stays positive definite: This flattens the search ellipsoid against canyon walls, preventing wasteful mutations into known high-loss regions. Jastrebski & Arnold 2006 measured speedups up to about 2× on ill-conditioned functions. Multimodal Restart Strategies: IPOP vs BIPOP Escaping deceptive local basins with automated population expansion schedules IPOP-CMA-ES doubles the population size after every restart. Larger populations increase global search power, smoothing over high-frequency local ripples. - Step-size collapse: - No best-fitness improvement across 10 generations - Condition number explosion: Where CMA-ES Pairs with Deep Learning & Creative AI Zero-gradient search in high-level representation and latent control spaces Encode learning rate schedules, dropout probabilities, normalization epsilons, and discrete layer topologies into a unified box. CMA-ES learns which parameter interactions matter without backprop through training loops. - Vectorized ask/tell loops for massive cluster parallelism. - IPOP/BIPOP restarts to escape poor initialization basins. Searching non-differentiable generative spaces: continuous cellular automata convolution kernels, latent prompt embeddings, and discrete tool-calling agent policies. - Optimize aesthetic scores, spatial structure metrics, or human feedback rewards. - Rank-based selection shrugs off warped reward scales; escaping local basins is the job of population sizing and IPOP/BIPOP restarts. - Full-covariance CMA-ES is practical up to a few thousand dimensions; larger latents call for sep-CMA-ES or low-rank variants. Continuous Morphodynamic Artificial Life Lenia Black-Box Physics Search Evolving growth-rule parameters μ, σ, Δt for soliton morphogenesis with CMA-ES; the convolution kernel itself stays fixed Continuous artificial life exists strictly inside chaotic, narrow parameter corridors. The update rule itself is smooth, but the fitness does a pattern survive? is a discontinuous functional of a chaotic rollout 18 steps per evaluation here , so gradients through it explode or vanish into uselessness. CMA-ES needs only rank comparisons, adapting its covariance ellipsoid to follow the razor-thin boundary of living emergence. Two Optimization Worlds: GECCO vs NeurIPS Why the black-box and deep-learning communities keep reinventing each other The world of Evolution Strategies, Estimation of Distribution Algorithms EDAs , and Kriging / Gaussian Process surrogates. Core Premise: Evaluations are precious, expensive, and opaque. Strength: Invariance to any monotone rescaling of the objective; tolerant of discontinuous, non-smooth landscapes. Benchmark Arena: Robotics controllers, CFD, structural FEA, aerodynamic design. The world of reverse-mode automatic differentiation, backpropagation, Adam/SGD, and end-to-end differentiable neural architectures. Core Premise: Differentiate billions of parameters with massive GPU throughput. Blind Spot: Struggles whenever simulators contain discrete jumps or non-smooth loops. Re-invention: Natural Evolution Strategies NES/xNES derive the same natural-gradient update CMA-ES approximates; OpenAI's ES for RL strips it down to a fixed isotropic Gaussian with a mean-only update. Both paradigms converge when they optimize parameter probability distributions with natural gradients under the Fisher Information metric. Bridging zero-order distribution updates with first-order automatic differentiation enables robust exploration across both physical engineering models and neural architectures.