Consensus scoring criteria for improving enrichment in virtual screening

MOTIVATION: Virtual screening of molecular compound libraries is a potentially powerful and inexpensive method for the discovery of novel lead compounds for drug development. The major weakness of virtual screening-the inability to consistently identify true positives (leads)-is likely due to our incomplete understanding of the chemistry involved in ligand binding and the subsequently imprecise scoring algorithms. It has been demonstrated that combining multiple scoring functions (consensus scoring) improves the enrichment of true positives. Previous efforts at consensus scoring have largely focused on empirical results, but they have yet to provide a theoretical analysis that gives insight into real features of combinations and data fusion for virtual screening. RESULTS: We demonstrate that combining multiple scoring functions improves the enrichment of true positives only if (a) each of the individual scoring functions has relatively high performance and (b) the individual scoring functions are distinctive. Notably, these two prediction variables are previously established criteria for the performance of data fusion approaches using either rank or score combinations. This work, thus, establishes a potential theoretical basis for the probable success of data fusion approaches to improve yields in in silico screening experiments. Furthermore, it is similarly established that the second criterion (b) can, in at least some cases, be functionally defined as the area between the rank versus score plots generated by the two (or more) algorithms. Because rank-score plots are independent of the performance of the individual scoring function, this establishes a second theoretically defined approach to determining the likely success of combining data from different predictive algorithms. This approach is, thus, useful in practical settings in the virtual screening process when the performance of at least two individual scoring functions (such as in criterion a) can be estimated as having a high likelihood of having high performance, even if no training sets are available. We provide initial validation of this theoretical approach using data from five scoring systems with two evolutionary docking algorithms on four targets, thymidine kinase, human dihydrofolate reductase, and estrogen receptors of antagonists and agonists. Our procedure is computationally efficient, able to adapt to different situations, and scalable to a large number of compounds as well as to a greater number of combinations. Results of the experiment show a fairly significant improvement (vs single algorithms) in several measures of scoring quality, specifically "goodness-of-hit" scores, false positive rates, and "enrichment". This approach (available online at http://gemdock.life. nctu.edu.tw/dock/download.php) has practical utility for cases where the basic tools are known or believed to be generally applicable, but where specific training sets are absent.     

Impact of scoring functions on enrichment in docking-based virtual screening: an application study on renin inhibitors

The docking program LigandFit/Cerius(2) has been used to perform shape-based virtual screening of databases against the aspartic protease renin, a target of determined three-dimensional structure. The protein structure was used in the induced fit binding conformation that occurs when renin is bound to the highly active renin inhibitor 1 (IC(50) = 2 nM). The scoring was calculated using several different scoring functions in order to get insight into the predictability of the magnitude of binding interactions. A database of 1000 diverse and druglike compounds, comprised of 990 members of a virtual database generated by using the iLib diverse software and 10 known active renin inhibitors, was docked flexibly and scored to determine appropriate scoring functions. All seven scoring functions used (LigScore1, LigScore2, PLP1, PLP2, JAIN, PMF, LUDI) were able to retrieve at least 50% of the active compounds within the first 20% (200 molecules) of the entire test database. A hit rate of 90% in the top 1.4% resulted using the quadruple consensus scoring of LigScore2, PLP1, PLP2, and JAIN. Additionally, a focused database was created with the iLib diverse software and used for the same procedure as the test database. Docking and scoring of the 990 focused compounds and the 10 known actives were performed. A hit rate of 100% in the top 8.4% resulted with use of the triple consensus scoring of PLP1, PLP2, and PMF. As expected, a ranking of the known active compounds within the focused database compared to the test database was observed. Adequate virtual screening conditions were derived empirically. They can be used for proximate docking and scoring application of compounds with putative renin inhibiting potency.     

How does consensus scoring work for virtual library screening? An idealized computer experiment

It has been reported recently that consensus scoring, which combines multiple scoring functions in binding affinity estimation, leads to higher hit-rates in virtual library screening studies. This method seems quite independent to the target receptor, the docking program, or even the scoring functions under investigation. Here we present an idealized computer experiment to explore how consensus scoring works. A hypothetical set of 5000 compounds is used to represent a chemical library under screening. The binding affinities of all its member compounds are assigned by mimicking a real situation. Based on the assumption that the error of a scoring function is a random number in a normal distribution, the predicted binding affinities were generated by adding such a random number to the "observed" binding affinities. The relationship between the hit-rates and the number of scoring functions employed in scoring was then investigated. The performance of several typical ranking strategies for a consensus scoring procedure was also explored. Our results demonstrate that consensus scoring outperforms any single scoring for a simple statistical reason: the mean value of repeated samplings tends to be closer to the true value. Our results also suggest that a moderate number of scoring functions, three or four, are sufficient for the purpose of consensus scoring. As for the ranking strategy, both the rank-by-number and the rank-by-rank strategy work more effectively than the rank-by-vote strategy.     

A <Emphasis Type="Italic">k</Emphasis>-nearest neighbor classification of hERG K<Superscript>+</Superscript> channel blockers

A series of 172 molecular structures that block the hERG K⁺ channel were used to develop a classification model where, initially, eight types of PaDEL fingerprints were used for k-nearest neighbor model development. A consensus model constructed using Extended-CDK, PubChem and Substructure count fingerprint-based models was found to be a robust predictor of hERG activity. This consensus model demonstrated sensitivity and specificity values of 0.78 and 0.61 for the internal dataset compounds and 0.63 and 0.54 for the external (PubChem) dataset compounds, respectively. This model has identified the highest number of true positives (i.e. 140) from the PubChem dataset so far, as compared to other published models, and can potentially serve as a basis for the prediction of hERG active compounds. Validating this model against FDA-withdrawn substances indicated that it may even be useful for differentiating between mechanisms underlying QT prolongation.     

Conditional probability: a new fusion method for merging disparate virtual screening results

This paper introduces a new consensus scoring approach for merging the results of different virtual screening methods based on conditional probabilities. The technique is experimentally evaluated using several ligand-based virtual screening methods and compared to two variations of the established Sum-rank fusion method where it performs as well or better than the Sum-rank methods. Our experiments confirm that consensus scoring increases the number of active compounds retrieved with respect to the best individual methods on average.     

Fingerprint scaling increases the probability of identifying molecules with similar activity in virtual screening calculations.

Results of systematic virtual screening calculations using a structural key-type fingerprint are reported for compounds belonging to 14 activity classes added to randomly selected synthetic molecules. For each class, a fingerprint profile was calculated to monitor the relative occupancy of fingerprint bit positions. Consensus bit patterns were determined consisting of all bits that were always set on in compounds belonging to a specific activity class. In virtual screening calculations, scale factors were applied to each consensus bit position in fingerprints of query molecules. This technique, called "fingerprint scaling", effectively increases the weight of consensus bit positions in fingerprint comparisons. Although overall prediction accuracy was satisfactory using unscaled calculations, scaling significantly increased the number of correct predictions but only slightly increased the rate of false positives. These observations suggest that fingerprint scaling is an attractive approach to increase the probability of identifying molecules with similar activity by virtual screening. It requires the availability of a series of related compounds and can be easily applied to any keyed fingerprint representation that associates bit positions with specific molecular features.     

Probability-based shotgun cross-linking sites analysis

We recently developed a shotgun tool for cross-linking sites analysis, X!Link, for the sensitive and high-throughput analysis of chemically cross-linked proteins or multiprotein complexes (J. Proteome Res. 2007, 6, 3908–3917). Here, we report a further development of the tool using a probability-based scoring system. It calculates explicit E-values, with which sensitive detection of the cross-links is possible with very low false positives, and now can be applied to moderate numbers of protein sequences. Most of the false positives in large scale analysis originate from partial matching where one side of the peptides is correctly matched while the other side is incorrectly matched. Additional E-values were calculated for each peptide and effectively minimized false positives from such partial matching. The usefulness of the new scoring system was demonstrated for a previously published dataset from a cross-linked cytochrome c protein, searching against a large database of equine protein sequences.     

Core trees and consensus fragment sequences for molecular representation and similarity analysis

A new type of molecular representation is introduced that is based on activity class characteristic substructures extracted from random fragment populations. Mapping of characteristic substructures is used to determine atom match rates in active molecules. Comparison of match rates of bonded atoms defines a hierarchical molecular fragmentation scheme. Active compounds are encoded as fragmentation pathways isolated from core trees. These paths are amenable to biological sequence alignment methods in combination with substructure-based scoring functions. From multiple core path alignments, consensus fragment sequences are derived that represent compound activity classes. Consensus fragment sequences weighted by increasing structural specificity can also be used to map molecules and search databases for active compounds.     

Maximum common binding modes (MCBM): consensus docking scoring using multiple ligand information and interaction fingerprints

Improving the scoring functions for small molecule-protein docking is a highly challenging task in current computational drug design. Here we present a novel consensus scoring concept for the prediction of binding modes for multiple known active ligands. Similar ligands are generally believed to bind to their receptor in a similar fashion. The presumption of our approach was that the true binding modes of similar ligands should be more similar to each other compared to false positive binding modes. The number of conserved (consensus) interactions between similar ligands was used as a docking score. Patterns of interactions were modeled using ligand receptor interaction fingerprints. Our approach was evaluated for four different data sets of known cocrystal structures (CDK-2, dihydrofolate reductase, HIV-1 protease, and thrombin). Docking poses were generated with FlexX and rescored by our approach. For comparison the CScore scoring functions from Sybyl were used, and consensus scores were calculated thereof. Our approach performed better than individual scoring functions and was comparable to consensus scoring. Analysis of the distribution of docking poses by self-organizing maps (SOM) and interaction fingerprints confirmed that clusters of docking poses composed of multiple ligands were preferentially observed near the native binding mode. Being conceptually unrelated to commonly used docking scoring functions our approach provides a powerful method to complement and improve computational docking experiments.     

Understanding false positives in reporter gene assays: in silico chemogenomics approaches to prioritize cell-based HTS data

High throughput screening (HTS) data is often noisy, containing both false positives and negatives. Thus, careful triaging and prioritization of the primary hit list can save time and money by identifying potential false positives before incurring the expense of followup. Of particular concern are cell-based reporter gene assays (RGAs) where the number of hits may be prohibitively high to be scrutinized manually for weeding out erroneous data. Based on statistical models built from chemical structures of 650 000 compounds tested in RGAs, we created "frequent hitter" models that make it possible to prioritize potential false positives. Furthermore, we followed up the frequent hitter evaluation with chemical structure based in silico target predictions to hypothesize a mechanism for the observed "off target" response. It was observed that the predicted cellular targets for the frequent hitters were known to be associated with undesirable effects such as cytotoxicity. More specifically, the most frequently predicted targets relate to apoptosis and cell differentiation, including kinases, topoisomerases, and protein phosphatases. The mechanism-based frequent hitter hypothesis was tested using 160 additional druglike compounds predicted by the model to be nonspecific actives in RGAs. This validation was successful (showing a 50% hit rate compared to a normal hit rate as low as 2%), and it demonstrates the power of computational models toward understanding complex relations between chemical structure and biological function.     

Consensus scoring with feature selection for structure-based virtual screening

The evaluation of ligand conformations is a crucial aspect of structure-based virtual screening, and scoring functions play significant roles in it. While consensus scoring (CS) generally improves enrichment by compensating for the deficiencies of each scoring function, the strategy of how individual scoring functions are selected remains a challenging task when few known active compounds are available. To address this problem, we propose feature selection-based consensus scoring (FSCS), which performs supervised feature selection with docked native ligand conformations to select complementary scoring functions. We evaluated the enrichments of five scoring functions (F-Score, D-Score, PMF, G-Score, and ChemScore), FSCS, and RCS (rank-by-rank consensus scoring) for four different target proteins: acetylcholine esterase (AChE), thrombin (thrombin), phosphodiesterase 5 (PDE5), and peroxisome proliferator-activated receptor gamma (PPARgamma). The results indicated that FSCS was able to select the complementary scoring functions and enhance ligand enrichments and that it outperformed RCS and the individual scoring functions for all target proteins. They also indicated that the performances of the single scoring functions were strongly dependent on the target protein. An especially favorable result with implications for practical drug screening is that FSCS performs well even if only one 3D structure of the protein-ligand complex is known. Moreover, we found that one can infer which scoring functions significantly enrich active compounds by using feature selection before actual docking and that the selected scoring functions are complementary.     

Identifying biologically active compound classes using phenotypic screening data and sampling statistics

Scoring the activity of compounds in phenotypic high-throughput assays presents a unique challenge because of the limited resolution and inherent measurement error of these assays. Techniques that leverage the structural similarity of compounds within an assay can be used to improve the hit-recovery rate from screening data. A technique is presented that uses clustering and sampling statistics to predict likely compound activity by scoring entire structural classes. A set of phenotypic assays performed against a commercially available compound library was used as a test set. Using the class-scoring technique, the resultant activity prediction scores were more reproducible than individual assay measurements, and class scoring recovered known active compounds more efficiently than individual assay measurements because class scoring had fewer false positives. Known biologically active compounds were recovered 87% of the time using class scores, suggesting a low false-negative rate that compared well to individual assay measurements. In addition, many weak and potentially novel classes of active compounds, overlooked by individual assay measurements, were suggested.     

Docking and scoring with ICM: the benchmarking results and strategies for improvement

Flexible docking and scoring using the internal coordinate mechanics software (ICM) was benchmarked for ligand binding mode prediction against the 85 co-crystal structures in the modified Astex data set. The ICM virtual ligand screening was tested against the 40 DUD target benchmarks and 11-target WOMBAT sets. The self-docking accuracy was evaluated for the top 1 and top 3 scoring poses at each ligand binding site with near native conformations below 2?? RMSD found in 91 and 95% of the predictions, respectively. The virtual ligand screening using single rigid pocket conformations provided the median area under the ROC curves equal to 69.4 with 22.0% true positives recovered at 2% false positive rate. Significant improvements up to ROC AUC?=?82.2 and ROC((2%))?=?45.2 were achieved following our best practices for flexible pocket refinement and out-of-pocket binding rescore. The virtual screening can be further improved by considering multiple conformations of the target.     

Enrichment of high-throughput screening data with increasing levels of noise using support vector machines, recursive partitioning, and laplacian-modified naive bayesian classifiers

High-throughput screening (HTS) plays a pivotal role in lead discovery for the pharmaceutical industry. In tandem, cheminformatics approaches are employed to increase the probability of the identification of novel biologically active compounds by mining the HTS data. HTS data is notoriously noisy, and therefore, the selection of the optimal data mining method is important for the success of such an analysis. Here, we describe a retrospective analysis of four HTS data sets using three mining approaches: Laplacian-modified naive Bayes, recursive partitioning, and support vector machine (SVM) classifiers with increasing stochastic noise in the form of false positives and false negatives. All three of the data mining methods at hand tolerated increasing levels of false positives even when the ratio of misclassified compounds to true active compounds was 5:1 in the training set. False negatives in the ratio of 1:1 were tolerated as well. SVM outperformed the other two methods in capturing active compounds and scaffolds in the top 1%. A Murcko scaffold analysis could explain the differences in enrichments among the four data sets. This study demonstrates that data mining methods can add a true value to the screen even when the data is contaminated with a high level of stochastic noise.     

Virtual screening of 4-anilinoquinazoline analogues as EGFR kinase inhibitors: importance of hydrogen bonds in the evaluation of poses and scoring functions

Virtual Screening (VS) is a computational technique that allows selection and ranking of possible hits from a library of compounds. We have carried out VS on 128 selected EGFR kinase inhibitors with GOLD and LigandFit. From the experimental crystal structure of the erlotinib-EGFR complex, three key hydrogen bonds were identified as responsible for anchoring the ligand in the active site. These are of the N-H...N, O(w)-H...N, and C-H...O types. Failure to include the hydrogen-bonded water molecule that forms the O(w)-H...N bond leads to incorrect results. Of the three interactions, the C-H...O formed by an activated C-H group is the best conserved. On the basis of the efficacy of these hydrogen bonds, the poses were classified into one of three categories: close, shifted, and misoriented. In the VS context, all three interactions need to be modeled correctly so that correct poses and affinities are obtained, and this happens in ligands of the close variety. Cross scoring wherein the poses from one software are input into another for scoring and consensus scoring wherein the scores from various software packages are weighted are also helpful in obtaining better agreements.     

In silico identification and evaluation of new Trypanosoma cruzi trypanothione reductase (TcTR) inhibitors obtained from natural products database of the Bahia semi-arid region (NatProDB)

Trypanosoma cruzi Trypanothione Reductase (TcTR) is one of the therapeutic targets studied in the development of new drugs against Chagas' disease. Due to its biodiversity, Brazil has several compounds of natural origin that were not yet properly explored in drug discovery. Therefore, we employed the Virtual Screening against TcTR aiming to discover new inhibitors from the Natural Products Database of the Bahia Semi-Arid region (NatProDB). This database has a wide chemical diversity favoring the discovery of new chemical entities. Subsequently, we analyzed the best docking conformations using self-organizing maps (AuPosSOM) aiming to verify their interaction sites at TcTR. Finally, the Pred-hERG, the Aggregator Advisor, the FAF-DRUGS and the pkCSM results allowed us to evaluate, respectively, the cardiotoxicity, aggregation capacity, presence of false positives (PAINS) and its toxicity. Thus, we selected three molecules that could be tested in in vitro assays in the hope that the computational results reported here would favor the development of new anti-chagasic drugs.     

New scoring functions for virtual screening from molecular dynamics simulations with a quantum-refined force-field (QRFF-MD). Application to cyclin-dependent kinase 2

A recently introduced new methodology based on ultrashort (50-100 ps) molecular dynamics simulations with a quantum-refined force-field (QRFF-MD) is here evaluated in its ability both to predict protein-ligand binding affinities and to discriminate active compounds from inactive ones. Physically based scoring functions are derived from this approach, and their performance is compared to that of several standard knowledge-based scoring functions. About 40 inhibitors of cyclin-dependent kinase 2 (CDK2) representing a broad chemical diversity were considered. The QRFF-MD method achieves a correlation coefficient, R(2), of 0.55, which is significantly better than that obtained by a number of traditional approaches in virtual screening but only slightly better than that obtained by consensus scoring (R(2) = 0.50). Compounds from the Available Chemical Directory, along with the known active compounds, were docked into the ATP binding site of CDK2 using the program Glide, and the 650 ligands from the top scored poses were considered for a QRFF-MD analysis. Combined with structural information extracted from the simulations, the QRFF-MD methodology results in similar enrichment of known actives compared to consensus scoring. Moreover, a new scoring function is introduced that combines a QRFF-MD based scoring function with consensus scoring, which results in substantial improvement on the enrichment profile.     

Virtual target screening: validation using kinase inhibitors

Computational methods involving virtual screening could potentially be employed to discover new biomolecular targets for an individual molecule of interest (MOI). However, existing scoring functions may not accurately differentiate proteins to which the MOI binds from a larger set of macromolecules in a protein structural database. An MOI will most likely have varying degrees of predicted binding affinities to many protein targets. However, correctly interpreting a docking score as a hit for the MOI docked to any individual protein can be problematic. In our method, which we term "Virtual Target Screening (VTS)", a set of small drug-like molecules are docked against each structure in the protein library to produce benchmark statistics. This calibration provides a reference for each protein so that hits can be identified for an MOI. VTS can then be used as tool for: drug repositioning (repurposing), specificity and toxicity testing, identifying potential metabolites, probing protein structures for allosteric sites, and testing focused libraries (collection of MOIs with similar chemotypes) for selectivity. To validate our VTS method, twenty kinase inhibitors were docked to a collection of calibrated protein structures. Here, we report our results where VTS predicted protein kinases as hits in preference to other proteins in our database. Concurrently, a graphical interface for VTS was developed.