Skip to content Skip to main navigation Report an accessibility issue

A scalable evolution strategy for high-dimensional black-box optimization and its application in reinforcement learning

Dr. Guannan Zhang
Research Staff
Oak Ridge National Lab
Friday, April 13, 2020 2:30-3:30 pm

Abstract: We developed an Evolution Strategy with Directional Gaussian Smoothing (DGS-ES) which exploits nonlocal searching to maximize/minimize high-dimensional non-convex black-box functions. The main contributions of this effort include (i) development of the a new DGS-gradient operator and its Gauss-Hermite estimator, which introduces, for the first time, an accurate nonlocal searching technique into the family of Evolution Strategy (ES). (ii) Theoretical analysis verifies that the scalability of the DGS-ES method, i.e., the number of iterations needed for convergence, is independent of the dimension for convex functions. (iii) Demonstration of the DGS-ES method on both high-dimensional non-convex benchmark optimization problems, as well as a real-world material design problem for rocket shell manufacture. (iv) Massive parallelization: the DGS-ES method is suited to be scaled up to a large number of parallel workers. All the function evaluations within each iteration can be simulated totally in parallel, and each worker only needs to return a scalar to the master, such that the communication cost among workers is minimal. Through several benchmark RL tasks demonstrated herein, we show that DGS-ES is highly scalable, possesses superior wall-clock time, and achieves competitive reward scores to other popular policy gradient and ES approaches.

Bio: Dr. Guannan Zhang is a Research Staff in Computational and Applied Mathematics Group at Oak Ridge National Laboratory (ORNL). He studied mathematics at Shandong University in China, receiving his Bachelor’s degree and Master’s degree in 2007 and 2009, respectively. Guannan earned his Ph.D. in computational science at Florida State University in 2012, under the supervision of Prof. Max Gunzburger. He joined ORNL in 2012 as the Householder fellow in the Computational and Applied Mathematics Group within the Computer Science and Mathematics Division. He has been holding a joint faculty appointment with the Department of Mathematics and Statistics at Auburn University since 2014. His research interests include high-dimensional approximation, uncertainty quantification, machine learning and artificial intelligence, stochastic optimization and control, numerical solution of stochastic differential equations, and model reduction for parametrized differential equations.