Development of hardware accelerator for molecular dynamics simulations: a computation board that calculates nonbonded interactions in cooperation with fast multipole method

Evaluation of long-range Coulombic interactions still represents a bottleneck in the molecular dynamics (MD) simulations of biological macromolecules. Despite the advent of sophisticated fast algorithms, such as the fast multipole method (FMM), accurate simulations still demand a great amount of computation time due to the accuracy/speed trade-off inherently involved in these algorithms. Unless higher order multipole expansions, which are extremely expensive to evaluate, are employed, a large amount of the execution time is still spent in directly calculating particle-particle interactions within the nearby region of each particle. To reduce this execution time for pair interactions, we developed a computation unit (board), called MD-Engine II, that calculates nonbonded pairwise interactions using a specially designed hardware. Four custom arithmetic-processors and a processor for memory manipulation ("particle processor") are mounted on the computation board. The arithmetic processors are responsible for calculation of the pair interactions. The particle processor plays a central role in realizing efficient cooperation with the FMM. The results of a series of 50-ps MD simulations of a protein-water system (50,764 atoms) indicated that a more stringent setting of accuracy in FMM computation, compared with those previously reported, was required for accurate simulations over long time periods. Such a level of accuracy was efficiently achieved using the cooperative calculations of the FMM and MD-Engine II. On an Alpha 21264 PC, the FMM computation at a moderate but tolerable level of accuracy was accelerated by a factor of 16.0 using three boards. At a high level of accuracy, the cooperative calculation achieved a 22.7-fold acceleration over the corresponding conventional FMM calculation. In the cooperative calculations of the FMM and MD-Engine II, it was possible to achieve more accurate computation at a comparable execution time by incorporating larger nearby regions.     

Scaling molecular dynamics beyond 100,000 processor cores for large-scale biophysical simulations

The growing interest in the complexity of biological interactions is continuously driving the need to increase system size in biophysical simulations, requiring not only powerful and advanced hardware but adaptable software that can accommodate a large number of atoms interacting through complex forcefields. To address this, we developed and implemented strategies in the GENESIS molecular dynamics package designed for large numbers of processors. Long-range electrostatic interactions were parallelized by minimizing the number of processes involved in communication. A novel algorithm was implemented for nonbonded interactions to increase single instruction multiple data (SIMD) performance, reducing memory usage for ultra large systems. Memory usage for neighbor searches in real-space nonbonded interactions was reduced by approximately 80%, leading to significant speedup. Using experimental data describing physical 3D chromatin interactions, we constructed the first atomistic model of an entire gene locus (GATA4). Taken together, these developments enabled the first billion-atom simulation of an intact biomolecular complex, achieving scaling to 65,000 processes (130,000 processor cores) with 1 ns/day performance. Published 2019. This article is a U.S. Government work and is in the public domain in the USA.     

Enhanced sampling method in molecular simulations using genetic algorithm for biomolecular systems

We propose a molecular simulation method using genetic algorithm (GA) for biomolecular systems to obtain ensemble averages efficiently. In this method, we incorporate the genetic crossover, which is one of the operations of GA, to any simulation method such as conventional molecular dynamics (MD), Monte Carlo, and other simulation methods. The genetic crossover proposes candidate conformations by exchanging parts of conformations of a target molecule between a pair of conformations during the simulation. If the candidate conformations are accepted, the simulation resumes from the accepted ones. While conventional simulations are based on local update of conformations, the genetic crossover introduces global update of conformations. As an example of the present approach, we incorporated genetic crossover to MD simulations. We tested the validity of the method by calculating ensemble averages and the sampling efficiency by using two kinds of peptides, ALA3 and (AAQAA)₃. The results show that for ALA3 system, the distribution probabilities of backbone dihedral angles are in good agreement with those of the conventional MD and replica-exchange MD simulations. In the case of (AAQAA)₃ system, our method showed lower structural correlation of α-helix structures than the other two methods and more flexibility in the backbone ψ angles than the conventional MD simulation. These results suggest that our method gives more efficient conformational sampling than conventional simulation methods based on local update of conformations. © 2018 Wiley Periodicals, Inc.     

Parallel Fock matrix construction program for molecular orbital calculation--specific computer with a hierarchical network

A parallel Fock matrix construction program for a hierarchical network has been developed on the molecular orbital calculation-specific EHPC system. To obtain high parallelization efficiency on the hierarchical network system, a multilevel dynamic load-balancing scheme was adopted, which provides equal load balance and localization of communications on a tree-structured hierarchical network. The parallelized Fock matrix construction routine was implemented into a GAMESS program on the EHPC system, which has a tree-structured hierarchical network. Benchmark results on a 63-processor system showed high parallelization efficiency even on the tree-structured hierarchical network.     

A scalable parallel Monte Carlo method for free energy simulations of molecular systems

We present a method of parallelizing flat histogram Monte Carlo simulations, which give the free energy of a molecular system as an output. In the serial version, a constant probability distribution, as a function of any system parameter, is calculated by updating an external potential that is added to the system Hamiltonian. This external potential is related to the free energy. In the parallel implementation, the simulation is distributed on to different processors. With regular intervals the modifying potential is summed over all processors and distributed back to every processor, thus spreading the information of which parts of parameter space have been explored. This implementation is shown to decrease the execution time linearly with added number of processors.     

Systematic benchmarking of large molecular dynamics simulations employing GROMACS on massive multiprocessing facilities

The influence of the total number of cores, the number of cores dedicated to Particle mesh Ewald (PME) calculation and the choice of single vs. double precision on the performance of molecular dynamic (MD) simulations in the size of 70,000 to 1.7 million of atoms was analyzed on three different high‐performance computing facilities employing GROMACS 4 by running about 6000 benchmark simulations. Small and medium sized systems scaled linear up to 64 and 128 cores, respectively. Systems with half a million to 1.2 million atoms scaled linear up to 256 cores. The best performance was achieved by dedicating 25% of the total number of cores to PME calculation. Double precision calculations lowered the performance by 30–50%. A database for collecting information about MD simulations and the achieved performance was created and is freely available online and allows the fast estimation of the performance that can be expected in similar environments. © 2010 Wiley Periodicals, Inc. J Comput Chem, 2011     

Assessment of biomolecular force fields for molecular dynamics simulations in a protein crystal

Different biomolecular force fields (OPLS‐AA, AMBER03, and GROMOS96) in conjunction with SPC, SPC/E and TIP3P water models are assessed for molecular dynamics simulations in a tetragonal lysozyme crystal. The root mean square deviations for the C_a atoms of lysozymes are about 0.1 to 0.2 nm from OPLS‐AA and AMBER03, smaller than 0.4 nm from GROMOS96. All force fields exhibit similar pattern in B‐factors, whereas OPLS‐AA and AMBER03 accurately reproduce experimental measurements. Despite slight variations, the primary secondary structures are well conserved using different force fields. Water diffusion in the crystal is approximately ten‐fold slower than in bulk phase. The directional and average water diffusivities from OPLS‐AA and AMBER03 along with SPC/E model match fairly well with experimental data. Compared to GROMOS96, OPLS‐AA and AMBER03 predict larger hydrophilic solvent‐accessible surface area of lysozyme, more hydrogen bonds between lysozyme and water, and higher percentage of water in hydration shell. SPC, SPC/E and TIP3P water models have similar performance in most energetic and structural properties, but SPC/E outperforms in water diffusion. While all force fields overestimate the mobility and electrical conductivity of NaCl, a combination of OPLS‐AA for lysozyme and the Kirkwood‐Buff model for ions is superior to others. As attributed to the steric restraints and surface interactions, the mobility and conductivity in the crystal are reduced by one to two orders of magnitude from aqueous solution. © 2009 Wiley Periodicals, Inc. J Comput Chem, 2010     

Performance evaluation of the zero‐multipole summation method in modern molecular dynamics software

The zero‐multiple summation method (ZMM) is a cutoff‐based method for calculating electrostatic interactions in molecular dynamics simulations, utilizing an electrostatic neutralization principle as a physical basis. Since the accuracies of the ZMM have been revealed to be sufficient in previous studies, it is highly desirable to clarify its practical performance. In this paper, the performance of the ZMM is compared with that of the smooth particle mesh Ewald method (SPME), where the both methods are implemented in molecular dynamics software package GROMACS. Extensive performance comparisons against a highly optimized, parameter‐tuned SPME implementation are performed for various‐sized water systems and two protein–water systems. We analyze in detail the dependence of the performance on the potential parameters and the number of CPU cores. Even though the ZMM uses a larger cutoff distance than the SPME does, the performance of the ZMM is comparable to or better than that of the SPME. This is because the ZMM does not require a time‐consuming electrostatic convolution and because the ZMM gains short neighbor‐list distances due to the smooth damping feature of the pairwise potential function near the cutoff length. We found, in particular, that the ZMM with quadrupole or octupole cancellation and no damping factor is an excellent candidate for the fast calculation of electrostatic interactions. © 2018 Wiley Periodicals, Inc.     

Rapid QM/MM approach for biomolecular systems under periodic boundary conditions: Combination of the density‐functional tight‐binding theory and particle mesh Ewald method

A quantum mechanical/molecular mechanical (QM/MM) approach based on the density‐functional tight‐binding (DFTB) theory is a useful tool for analyzing chemical reaction systems in detail. In this study, an efficient QM/MM method is developed by the combination of the DFTB/MM and particle mesh Ewald (PME) methods. Because the Fock matrix, which is required in the DFTB calculation, is analytically obtained by the PME method, the Coulomb energy is accurately and rapidly computed. For assessing the performance of this method, DFTB/MM calculations and molecular dynamics simulation are conducted for a system consisting of two amyloid‐β(1‐16) peptides and a zinc ion in explicit water under periodic boundary conditions. As compared with that of the conventional Ewald summation method, the computational cost of the Coulomb energy by utilizing the present approach is drastically reduced, i.e., 166.5 times faster. Furthermore, the deviation of the electronic energy is less than . © 2016 Wiley Periodicals, Inc.     

nMoldyn 3: Using task farming for a parallel spectroscopy‐oriented analysis of molecular dynamics simulations

We present a new version of the program package nMoldyn, which has been originally developed for a neutron‐scattering oriented analysis of molecular dynamics simulations of macromolecular systems (Kneller et al., Comput. Phys. Commun. 1995, 91, 191) and was later rewritten to include in‐depth time series analyses and a graphical user interface (Rog et al., J. Comput. Chem. 2003, 24, 657). The main improvement in this new version and the focus of this article are the parallelization of all the analysis algorithms for use on multicore desktop computers as well as distributed‐memory computing clusters. The parallelization is based on a task farming approach which maintains a simple program structure permitting easy modification and extension of the code to integrate new analysis methods. © 2012 Wiley Periodicals, Inc.     

Three pillars for achieving quantum mechanical molecular dynamics simulations of huge systems: Divide‐and‐conquer,density‐functional tight‐binding,and massively parallel computation

The linear‐scaling divide‐and‐conquer (DC) quantum chemical methodology is applied to the density‐functional tight‐binding (DFTB) theory to develop a massively parallel program that achieves on‐the‐fly molecular reaction dynamics simulations of huge systems from scratch. The functions to perform large scale geometry optimization and molecular dynamics with DC‐DFTB potential energy surface are implemented to the program called DC‐DFTB‐K. A novel interpolation‐based algorithm is developed for parallelizing the determination of the Fermi level in the DC method. The performance of the DC‐DFTB‐K program is assessed using a laboratory computer and the K computer. Numerical tests show the high efficiency of the DC‐DFTB‐K program, a single‐point energy gradient calculation of a one‐million‐atom system is completed within 60 s using 7290 nodes of the K computer. © 2016 Wiley Periodicals, Inc.     

Comparison of the accuracy of periodic reaction field methods in molecular dynamics simulations of a model liquid crystal system

A periodic reaction field (PRF) method is a technique to estimate long‐range interactions. The method has the potential to effectively reduce the computational cost while maintaining adequate accuracy. We performed molecular dynamics (MD) simulations of a model liquid‐crystal system to assess the accuracy of some variations of the PRF method in low‐charge‐density systems. All the methods had adequate accuracy compared with the results of the particle mesh Ewald (PME) method, except for a few simulation conditions. Furthermore, in all of the simulation conditions, one of the PRF methods had the same accuracy as the PME method. © 2015 Wiley Periodicals, Inc.     

Ab initio molecular dynamics simulations study on initial decompositions of β‐HMX at high temperature coupled with high pressures

We performed ab initio molecular dynamics simulations to investigate initial decomposition mechanisms and subsequent chemical processes of β‐HMX (cyclotetramethylene tetranitramine) (octahydro‐1,3,5,7‐tetranitro‐1,3,5,7‐tetrazocine) crystals at high temperature coupled with high pressures. It was found that the initial decomposition step is the simultaneous C–H and N–NO₂ bond cleavage at 3,500 K. When the pressure (1–10 GPa) is applied, the first reaction steps are primarily the C–N and C–H bond fission at 3,500 K. The C–H bond cleavage is a triggering decomposition step of the HMX crystals at 3,500 K coupled with 16 GPa. This indicates that the C–H bonds are much easier to be broken and the hydrogen radicals are much more active. The applied pressures (1–10 GPa) accelerate the decompositions of HMX at 3,500 K. The decomposition pathways and time evolution of the main chemical species demonstrate that the temperature is the foremost factor that affects the decomposition at high pressures (1–10 GPa). However, the decomposition of HMX is dependent on both the temperature (3,500 K) and the pressure (16 GPa). This work will enrich the knowledge of the decompositions of condensed energetic materials under extreme conditions.     

Force-field development and molecular dynamics simulations of ferrocene-peptide conjugates as a scaffold for hydrogenase mimics

The increasing importance of hydrogenase enzymes in the new energy research field has led us to examine the structure and dynamics of potential hydrogenase mimics, based on a ferrocene-peptide scaffold, using molecular dynamics (MD) simulations. To enable this MD study, a molecular mechanics force field for ferrocene-bearing peptides was developed and implemented in the CHARMM simulation package, thus extending the usefulness of the package into peptide-bioorganometallic chemistry. Using the automated frequency-matching method (AFMM), optimized intramolecular force-field parameters were generated through quantum chemical reference normal modes. The partial charges for ferrocene were derived by fitting point charges to quantum-chemically computed electrostatic potentials. The force field was tested against experimental X-ray crystal structures of dipeptide derivatives of ferrocene-1,1'-dicarboxylic acid. The calculations reproduce accurately the molecular geometries, including the characteristic C2-symmetrical intramolecular hydrogen-bonding pattern, that were stable over 0.1 micros MD simulations. The crystal packing properties of ferrocene-1-(D)alanine-(D)proline-1'-(D)alanine-(D)proline were also accurately reproduced. The lattice parameters of this crystal were conserved during a 0.1 micros MD simulation and match the experimental values almost exactly. Simulations of the peptides in dichloromethane are also in good agreement with experimental NMR and circular dichroism (CD) data in solution. The developed force field was used to perform MD simulations on novel, as yet unsynthesized peptide fragments that surround the active site of [Ni-Fe] hydrogenase. The results of this simulation lead us to propose an improved design for synthetic peptide-based hydrogenase models. The presented MD simulation results of metallocenes thereby provide a convincing validation of our proposal to use ferrocene-peptides as minimal enzyme mimics.     

On the achievement of high fidelity and scalability for large‐scale diagonalizations in grid‐based DFT simulations

Recent advance in high performance computing (HPC) resources has opened the possibility to expand the scope of density functional theory (DFT) simulations toward large and complex molecular systems. This work proposes a numerically robust method that enables scalable diagonalizations of large DFT Hamiltonian matrices, particularly with thousands of computing CPUs (cores) that are usual these days in terms of sizes of HPC resources. The well‐known Lanczos method is extensively refactorized to overcome its weakness for evaluation of multiple degenerate eigenpairs that is the substance of DFT simulations, where a multilevel parallelization is adopted for scalable simulations in as many cores as possible. With solid benchmark tests for realistic molecular systems, the fidelity of our method are validated against the locally optimal block preconditioned conjugated gradient (LOBPCG) method that is widely used to simulate electronic structures. Our method may waste computing resources for simulations of molecules whose degeneracy cannot be reasonably estimated. But, compared to LOBPCG method, it is fairly excellent in perspectives of both speed and scalability, and particularly has remarkably less (< 10%) sensitivity of performance to the random nature of initial basis vectors. As a promising candidate for solving electronic structures of highly degenerate systems, the proposed method can make a meaningful contribution to migrating DFT simulations toward extremely large computing environments that normally have more than several tens of thousands of computing cores.     

Theory for trivial trajectory parallelization of multicanonical molecular dynamics and application to a polypeptide in water

Trivial trajectory parallelization of multicanonical molecular dynamics (TTP-McMD) explores the conformational space of a biological system with multiple short runs of McMD starting from various initial structures. This method simply connects (i.e., trivially parallelizes) the short trajectories and generates a long trajectory. First, we theoretically prove that the simple trajectory connection satisfies a detailed balance automatically. Thus, the resultant long trajectory is regarded as a single multicanonical trajectory. Second, we applied TTP-McMD to an alanine decapeptide with an all-atom model in explicit water to compute a free-energy landscape. The theory imposes two requirements on the multiple trajectories. We have demonstrated that TTP-McMD naturally satisfies the requirements. The TTP-McMD produces the free-energy landscape considerably faster than a single-run McMD does. We quantitatively showed that the accuracy of the computed landscape increases with increasing the number of multiple runs. Generally, the free-energy landscape of a large biological system is unknown a priori. The current method is suitable for conformational sampling of such a large system to reduce the waiting time to obtain a canonical ensemble statistically reliable.     

An HPLC method for the quantitation of 3‐pentylbenzo[c]thiophen‐1(3H)‐one in dog plasma and its application to a pharmacokinetic study

A simple, sensitive and reproducible high‐performance liquid chromatography (HPLC) assay method was developed for the estimation of 3‐pentylbenzo[c]thiophen‐1(3H)‐one (S₅), a potential anti‐ischemic stroke agent, in dog plasma. The analytical procedure involves protein precipitation of S₅ and nobiletin (internal standard) from dog plasma with acetonitrile. Chromatographic separation was achieved on Sapphire C18 analytical column with methanol–water (80:20, v/v) as mobile phase. The eluate was monitored using a UV detector set at 260 nm. The calibration curves were linear over the range of 0.2–20 µg/mL. Absolute recoveries of S₅ were 79.2–86.1% from dog plasma. The intra‐ and inter‐day relative standard deviation precisions were <7 and 5%, respectively. The method was successfully applied to the pharmacokinetic study of S₅ in beagle dogs. Copyright © 2014 John Wiley & Sons, Ltd.     

Ca2+‐complex stability of GAPAGPLIVPY peptide in gas and aqueous phase,investigated by affinity capillary electrophoresis and molecular dynamics simulations and compared to mass spectrometric results

Strong, sequence‐specific gas‐phase bindings between proline‐rich peptides and alkaline earth metal ions in nanoESI‐MS experiments were reported by Lehmann et al. (Rapid Commun. Mass Spectrom. 2006, 20, 2404–2410), however its relevance for physiological‐like aqueous phase is uncertain. Therefore, the complexes should also be studied in aqueous solution and the relevance of the MS method for binding studies be evaluated. A mobility shift ACE method was used for determining the binding between the small peptide GAPAGPLIVPY and various metal ions in aqueous solution. The findings were compared to the MS results and further explained using computational methods. While the MS data showed a strong alkaline earth ion binding, the ACE results showed nonsignificant binding. The proposed vacuum state complex also decomposed during a molecular dynamic simulation in aqueous solution. This study shows that the formed stable peptide–metal ion adducts in the gas phase by ESI‐MS does not imply the existence of analogous adducts in the aqueous phase. Comparing peptide–metal ion interaction under the gaseous MS and aqueous ACE conditions showed huge difference in binding behavior.     

Efficient estimation of binding free energies between peptides and an MHC class II molecule using coarse‐grained molecular dynamics simulations with a weighted histogram analysis method

We estimate the binding free energy between peptides and an MHC class II molecule using molecular dynamics (MD) simulations with the weighted histogram analysis method (WHAM). We show that, owing to its more thorough sampling in the available computational time, the binding free energy obtained by pulling the whole peptide using a coarse‐grained (CG) force field (MARTINI) is less prone to significant error induced by inadequate‐sampling than using an atomistic force field (AMBER). We further demonstrate that using CG MD to pull 3–4 residue peptide segments while leaving the remaining peptide segments in the binding groove and adding up the binding free energies of all peptide segments gives robust binding free energy estimations, which are in good agreement with the experimentally measured binding affinities for the peptide sequences studied. Our approach thus provides a promising and computationally efficient way to rapidly and reliably estimate the binding free energy between an arbitrary peptide and an MHC class II molecule. © 2017 Wiley Periodicals, Inc.