DrugMint: a webserver for predicting and designing of drug-like molecules

Analysis of dataset

Principal Component Analysis (PCA)

We used the principal component analysis (PCA) for computing the variance among the experimental and the approved drugs [23]. As shown in Figure 1, the variance decreased significantly up to the PC-15. Afterwards, it remained more or less constant. The variance between PC-1 and PC-2 for the whole dataset was 15.76% and 8.91% respectively [Figure 2]. These results clearly indicated that the dataset was highly diverse for developing a prediction model.

Substructure fragment analysis

To explore the hidden information, the dataset was further analyzed using SubFP, MACCS keys based fingerprints using the formula given below;

$\mathit{Frequency}\mathit{of}a\mathit{fragment}=\left(N_{\mathit{fragment}\_\mathit{class}}*N_{\mathit{total}}/N_{\mathit{fragment}\_\mathit{total}}*N_{\mathit{class}}\right)$

Where N_{fragment_class} is the number of fragments present in that class (approved/experimental); N_total is the total number of molecules studied (approved + experimental); N_{fragment_total} is the total number of fragments in all molecules (approved + experimental); N_class is the number of molecules in that class (approved/experimental).

Our analysis suggested that some of the substructure fragments were not preferred in the approved drugs. The substructure-based analysis suggested that primary alcohol, phosphoric monoester, diester and mixed anhydride were non-preferable functional groups that were present in the experimental drugs with higher frequency [Table 1]. Similarly, MACCS keys 66, 112, 122, 138, 144, and 150 were highly desirable and present with higher frequency in the approved drugs [Table 2, Additional file 1: Table-S1 and Figure 3]. Therefore, while designing new drug-like molecule in the future, the exclusion of SubFP fingerprints and the inclusion of certain MACCS keys might increase the probability of designing a better molecule.

SubFP	Description	Approved		Experimental		Frequency difference
		Nfrag_aprd	Faprd	Nfrag_exp	Fexp	(Faprd-Fexp)
SubFP13	Primary_alcohol	139	0.69	539	1.13	-(0.44)
SubFP41	1-2Diol	67	0.31	663	1.29	-(0.98)
SubFP48	Aldehyde	4	0.16	79	1.35	-(1.19)
SubFP126	Alpha_hydorxyacid	1	0.07	50	1.39	-(1.32)
SubFP224	Sulfenic_derivatives	23	0.59	108	1.17	-(0.58)
SubFP237	Phosphoric_monoester	5	0.06	289	1.4	-(1.34)
SubFP238	Phosphoric_diester	3	0.2	47	1.33	-(1.13)
SubFP246	Phosphoric_acid_derivatives	16	0.1	506	1.38	-(1.28)
SubFP281	Sugar_pattern_1	71	0.34	627	1.28	-(0.94)
SubFP291	Mixed_anhydride	3	0.05	214	1.4	-(1.35)

^aN_{frag_aprd}: number of fragment in approved class, N_{frag_exp}: number of fragment in experimental class, F_aprd: frequency of particular fragment in approved class, F_exp: frequency of particular fragment in experimental class, negative sign indicates that these fingerprints are not preferable in approved drug.

Fingerprint	Accuracy (%)	Approved		Experimental		Frequency difference
		Nfrag_aprd	Faprd	Nfrag_exp	Fexp	(Faprd- Fexp)
MACCS112	66.35	1288	1.58	1473	0.76	0.82
MACCS122	66.66	1105	1.57	1276	0.76	0.81
MACCS144	69.21	972	1.64	1027	0.73	0.91
MACCS66	73.97	557	1.98	395	0.59	1.39
MACCS150	57.57	1227	1.36	1812	0.85	0.52
MACCS138	65.91	910	1.52	1115	0.78	0.74

^aN_{frag_aprd}: number of fragment in approved class, N_{frag_exp}: number of fragment in experimental class, F_aprd: frequency of particular fragment in approved class, F_exp: frequency of particular fragment in experimental class.

Classification models

In order to evaluate the performance of different fingerprints, we have developed various models on different sets of descriptors that were calculated by PaDEL software. Separate models were developed on fingerprints selected using attribute selection modules rm-useless and CfsSubsetEval of Weka.

Fingerprints based models

The initially developed models based on Estate, PubChem, Extended, FingerPrinter, GraphsOnly, Substructure finger, Substructure count, Klekota-count, Klekota-fingerprint showed nearly equal performance with MCC value in the range of 0.5 to 0.6 [Table 3]. However, the models developed using 159 MACCS keys, achieve maximum MCC 0.77 with accuracy 89.96% [Table 3, Figure 4]. In addition to that, we have also applied Monte-Carlo (MC) approach by generating 30 times training and testing dataset for five-fold cross-validation. We have observed that these results were more or less same with previously used five-fold cross-validation results having average 87.88%/90.36% sensitivity/ specificity, 89.63% accuracy with MCC value 0.76 (Additional file 1: Table-S2).

Fingerprints	Selection-algorithm	Descriptors	Threshold	Sensitivity	Specificity	Accuracy	MCC	AUC
Estate	rm-useless	52	0	70.23	80.6	77.53	0.49	0.82
	cfsSubsetEval	9	0.3	70.9	71.68	71.45	0.4	0.77
Extended	rm-useless	1012	0	62.44	92.51	83.62	0.59	0.86
	cfsSubsetEval	25	−0.4	60.13	85.93	78.3	0.47	0.79
Fingerprinter	rm-useless	1024	0	61.99	92.36	83.37	0.58	0.86
	cfsSubsetEval	40	−0.2	63.62	85.84	79.27	0.5	0.81
GraphsOnly	rm-useless	1024	0	67.78	86.28	80.8	0.54	0.85
	cfsSubsetEval	43	0	69.86	73.71	72.57	0.41	0.78
Pubchem	rm-useless	704	0	63.85	92.11	83.75	0.59	0.87
	cfsSubsetEval	27	0.4	66.59	79.76	75.86	0.45	0.8
MACCS	rm-useless	159	0	88.42	90.61	89.96	0.77	0.95
	cfsSubsetEval	10	0	89.83	81.72	84.12	0.67	0.87
Substr-count	rm-useless	192	0	93.1	87.84	89.39	0.77	0.95
	cfsSubsetEval	16	−0.3	84.71	78.32	80.21	0.59	0.88
Sub-finger	rm-useless	192	0	76.32	78.45	77.82	0.52	0.84
	cfsSubsetEval	18	0	50.71	84.44	74.46	0.37	0.74
Klekota-count	rm-useless	2273	−0.2	63.92	90.42	82.58	0.57	0.85
	cfsSubsetEval	57	0	72.31	80.82	78.3	0.51	0.82
Klekota-finger	rm-useless	2273	0	61.84	92.89	83.7	0.59	0.86
	cfsSubsetEval	51	0	53.75	91.98	80.67	0.51	0.81

PCA based model

Hybrid models

Next, by combining the descriptors of CfsSubsetEval module for each fingerprint, a hybrid model (Hybrid-4) was developed which showed accuracy up to 90.07% with a MCC value of 0.78 [Table 5]. Finally, a hybrid (Hybrid-5) model on 22 descriptors was obtained upon further reducing these descriptors (296) by CfsSubsetEval module and it resulted in a slight decrease in MCC value to 0.7 with a significant reduction in the number of descriptors.

Performance on validation dataset

Performance on independent dataset

We tested our MACCS (ISIS) keys based model on the independent dataset and achieved 84% sensitivity, 38.92% specificity with accuracy value of 41.15%. These results also indicated that ~61% of the molecules present in our independent dataset have the potential to be in the approved category in future. Recently, twenty-one drugs were approved in the DrugBank v3.0, which was not classified as approved in the earlier release. Interestingly, all these compounds were classified in the ‘drug-like’ class by our model and this result clearly exemplified the performance of our model. Together, these results also indicated that our model could be very useful in the prediction of drug-like properties of a given compound in advance.

Screening of databases

We predicted drug-like potential of molecules in three major databases ChEMBL, ZINC and directory of useful decoys (DUD). The screening of 10384763 compounds from ZINC database showed that 78.33% (8134753 molecules) among them have the potential to be drug-like (http://crdd.osdd.net/oscadd/drugmint/data/zinc.csv). Similarly, ChEMBL dataset contained 1251913 molecules, only 72.43% (906791 molecules) were predicted to have drug-like properties (http://crdd.osdd.net/oscadd/drugmint/data/chembl.csv). Finally, our software predicted ~62% and ~64% of the compounds that are present in active and decoys datasets respectively to be drug-like [Additional file 2]. These results indicated that despite the growth of a large number of chemicals showing pharmacological activity in a particular condition, not all molecules have potential for satisfying the drug-like properties.

Web server

The major drawback of most of chemo-informatics studies is that they are mainly based on commercial software packages. This is the reason most of the predictive studies described in literature are not available for public use in the form of software or web server. In order to overcome this drawback, we have used freely available software and achieved results comparable to those that have used commercial software. Our study is implemented in the form of a webserver without any restriction. In this server, we have provided the facility to design, screen and predict the drug-likeness score of chemical compounds. The screening results of ZINC and ChEMBL library are also provided in the option of database search. In order to provide this free service to the community, we have developed “drugmint” (http://crdd.osdd.net/oscadd/drugmint) a user-friendly webserver for discriminating the approved drug from the experimental drugs. This server allows users to interactively draw/modify a molecule using a Marvin applet [24]. This server is installed on Linux (Red Hat) operating system. The common gateway interface (CGI) scripts of “drugmint” are written using PERL version 5.03.

Dataset source

Descriptors of molecules

Selection of descriptors

It has been shown in previous studies that all descriptors are not relevant [27]. Thus, the selection of descriptors is a crucial step for developing any kind of prediction model [28, 29]. In this study, we used two modules of Weka i) Remove Useless (rm-useless) and ii) CfsSubsetEval with best-fit algorithm [30]. In case of rm-useless, all those descriptors, which either varies too much or variation is negligible, have been removed. The CfsSsubsetEval module of Weka is a rigorous algorithm; it selects only those features or descriptors that have high correlation with class/activity and very less inter-correlation.

Cross-validation techniques

Leave one out cross-validation (LOOCV) is a preferred technique to evaluate the performance of a model. This technique is time consuming and CPU intensive particularly when dataset is large. In this study, we have used five-fold cross-validation technique to reduce the computational time for developing and evaluating our models. In this technique, the whole data set is randomly divided into five sets of similar size, four sets are used for training and remaining set for testing. This process is repeated five times in such a way that each set is used only once for testing. Overall performance is computed on the whole dataset after repeating the aforesaid process five times.

Model development

In this study, we have developed Support Vector Machine (SVM) based models for prediction of drug-like molecules using SVM^light software package. SVM is based on the statistical and optimization theory and it handles complex structural features, and allows users to choose a number of parameters and kernels (e.g. linear, polynomial, radial basis function, and sigmoid) or any user-defined kernel. This software can be downloaded freely from http://www.cs.cornell.edu/People/tj/svm_light/.

Evaluation parameters

where TP and TN are the number of truly or correctly predicted positive (approved) and negative (experimental) drugs, respectively. FP and FN are the number of false or wrongly predicted approved and experimental drugs, respectively. Matthew’s correlation coefficient (MCC) is considered to be the most robust parameter of any class prediction method. We have also used a threshold-independent parameter called receiver-operating curve (ROC) for evaluating performance of our models.

Reviewer number 1: Dr Robert Murphy

Comment-1: This manuscript describe a fairly simply design of a machine learning system for predicting whether a chemical structure is similar to previously approved drugs. It describes a web server to provide predictions about new structures.

The manuscript does not provide sufficient discussion of relevant prior work and quantitative comparison with other published approaches for which code is available (e.g., Bickerton et al. 2012). Approaches such as features reflecting drug dynamics (e.g., Vistoli et al. (2008) Drug Discovery Today 13:285–294 (doi:10.1016/j.drudis.2007.11.007) are also not discussed.

Response: In the revised version, we have discussed the previous studies as suggested by reviewer. After getting comments from the reviewer, we evaluate performance of QED model on our datasets, QED correctly predict 44.8% (sensitivity) approved and 81.28% (specificity) experimental drugs. While on independent dataset, it shows only 40% sensitivity and 52.5% specificity. QED (Bickerton et al. 2012) perform poor on our dataset because it is developed for predicting oral drug-likeness of a molecule. The high sensitivity and specificity of our models described in this study implies its usefulness in predicting drug-likeness of a molecule.

Comment-2: There is a potentially serious concern with the validity of the results due to the fact that the experimental design may result in overfitting. Even though cross-validation was used internally for combinations of features and learners to evaluate predictive accuracies, when these results are subsequently used to make decisions (such as which features to use) it compromises any conclusions from further analysis of the same training and testing data. A related problem may also arise from maximization of ROC area when some of the experimental drugs may indeed be drug-like. These concerns were shown to be warranted because the final evaluation using an independent dataset showed much lower accuracy. However, it is somewhat encouraging that twenty-one molecules in the test set that were recently approved as drugs were classified as “drug-like” by the authors’ model.

Response: We are thankful to reviewer for this valuable comment. In order to further validate our prediction model, we used Monte-Carlo approach where we randomly create training and testing datasets 30 times and compute average performance. We achieved sensitivity 87.88%, specificity 90.36% and accuracy 89.63% when evaluated using Monte-Carlo approach. The result for every set is provided in supplementary document (Additional file 1: Table-S2) in the form of sensitivity, specificity, accuracy and MCC along with their mean and standard deviation. These results were more or less same to the previous five fold results. The result indicates that our models are not over-fitted and will be useful in real scenario.

Comment-3: The web server model does not seem appropriate for the primary use case, which is envisaged to be making predictions for users with novel structures. Since users may wish to keep their structures private, an open source approach would be strongly preferable to a public server. This would secure use of the system and also permit inspection and modification of the methods used.

Response: We are thankful for this suggestion. We understand the limitation of the webserver used for prediction. In order to facilitate and for the sake of user privacy, we developed a standalone version of this software, which is available for download from http://osddlinux.osdd.net, now user can run our software on their local machine.

Additional comment-1: The author list contains “Open Source Drug Discovery Consortium” which is not a person and is not mentioned elsewhere in the manuscript.

Response: We are thankful for this comment. In the revised version, we have acknowledged the Open Source Drug Discovery Consortium instead of authors list.

Additional comment-2: The abstract refers to screening but the manuscript does not describe any screening results.

Response: The authors are thankful for this suggestion. In the revised manuscript, we have provided the detailed of chemical libraries and their screening results under the paragraph screening of databases.

Quality of written English: Needs some language corrections before being published.

Response: We have corrected the language in the revised manuscript.

Reviewer number 2: Prof Difei Wang (nominated by Dr Yuriy Gusev)

In general, this is an interesting work and it is important to predict drug-like molecules using various types of molecular fingerprints. However, I do have some questions about the manuscript.

Comment-1: On page7, the authors stated that “Similarly, MACCS fingerprint elements 112, 122, 144, and 150 were highly desirable and present with higher frequency in the approved drugs [Table 2, Figure 3]”. How to interpret this observation? What are the definition of MACCS-144 and −150 etc.? It will be very useful if the authors can clearly explain what are these features. Also, MACCS-66 is missing here but it does show up in the Table. Is there any reason to exclude MACCS-66 here?

Comment-2: What is the score cutoff value for drug like and non drug like molecules for database screening results? What are the meaning of “drug like, low”, “drug like, high” and “non drug like, low”? What false-positive rate do we expect here?

Response: The authors are thankful for this comment. In this study, we have used a threshold value 0 for discrimination of the approved and experimental drugs.

False positive rate has been calculated via 30 times shuffling the dataset in five fold cross-validation and the average value of FPR is 9.64% (Additional file 2: Table-S2).

Comment-3: How many distinct structural families in drugbank3.0? How structurally diverse of this dataset? Are there many drugs having similar structures? If the answer is yes, will it bias the fingerprint selection and model creation?

Response: We are thankful for this valuable comment. After getting this comment, we analyzed the structural family of drugs in drugbank3.0 and found that at present these were classified into 233 different families (http://www.drugbank.ca/drug_classes). This clearly shows the dataset is highly diverse and suitable for model development.

Comment-4: I tried the example on the web server. But it seems slow and could not give me the result. Is this server really functional?

Response: We are thankful to the reviewer for this comment. Now, the server is completely functional.

Comment-5: Will it possible to have a standalone version of the web server? It will be great if there is a standalone version available to the community.

Response: We are thankful for such a nice suggestion. To improve the visibility of this work, we have developed a standalone version of this software. This is available to the users at http://osddlinux.osdd.net.

Comment-6: On page 1, "can predict drug-likeness of molecules with precession." Is "precession" a typo?

Response: We are thankful to the reviewer for pointing out this typo error. In the revised version, we have corrected this mistake and also take care of any other grammatical error.

Comment-7: I am not sure if this topic is suitable for this computational biology-centric journal. Maybe, this work is more suitable for publishing in journals like BMC.

Response: We are thankful for this suggestion and we think this kind of work is well suited for this journal.

Quality of written English: Acceptable

Reviewer number 3: Mr Ahmet Bakan (nominated by Prof James Faeder)

Comment-1: The authors developed various classification models using an exhaustive set of chemical fingerprints for discriminating approved drugs from experimental drugs and made these models available via a web server. In the past years, many newly approved drug molecules are breaking the widely accepted rule of 5 for drug-likeness, this improving and updating methods for calculating drug-likeness is an important problem. However, I don't understand why authors developed models that discriminate "approved" drugs from "experimental" drugs. Experimental drugs are molecules that are under investigation. Being experimental does not meet the compound is not drug-like, so any model that discriminates approved from experimental does not have any value. The exhaustive approach would be valuable if models were developed to discriminate drug-like, safe compounds from potentially toxic, non-drug-like compounds.

Response: We completely agreed with the reviewer comment. Although, studies have been done previously with focused towards the discrimination of drug-like molecules from non-drug-like ones. But most of these were based on the use of commercial dataset like MDDR, CMC as drug-like and ACD as non drug-like dataset. Thus, availability of the dataset is the major issue. In contrast, our method is an attempt to discriminate two closely related drug-like molecules. This will be an advance step in drug design process because despite the in vitro drug-like properties, many drugs failed in clinical trial (experimental stage). Thus, it is very important to discriminate these two classes of molecules. This is the only dataset that is available for public use and will be an excellent asset for development of public domain servers.

Quality of written English: Not suitable for publication unless extensively edited

Response: We are thankful to the reviewer for this comment. In the revised version, we have tried our best to improve quality of English in revised version of manuscript. Hopefully, the revised version will be suitable for publication.

Reviewer number 1: Dr Robert Murphy

The authors did not respond adequately to my concern about overfitting. By using the results from cross-validation to make choices (such as which features to use), the expected accuracy of the system so configured is no longer the cross-validation accuracy for that configuration. Simply adding more cross-validation trials does not address the issue. The problem may be clarified by considering that some combination of features and model parameters will optimize performance on any finite dataset but that the same combination may not be optimal for another finite dataset even if chosen from the same underlying distribution. Optimization of these choices does not allow the accuracy to be estimate for the new dataset. The point is that in order for cross-validation to be used to estimate future performance, all choices must be made using the training set only. The observation that the performance on the independent dataset (from DrugBank v3.0) was significantly worse suggests that the two datasets may have been drawn from different distributions (likely) but also that the cross-validation accuracy from the original dataset was an overestimate.

Response: After getting above comments on our revised version, we recheck reviewers comment and our previous response. We realize that we misunderstood comments, this is the reason we make more cross-validation trials. We agree with reviewers that we perform feature selection from whole dataset so there is biasness in feature selection. In this version of manuscript, we also evaluated performance of our models to avoid the ambiguity of biasness. We randomly picked 20% of the data from the whole dataset and called this dataset as validation dataset (for detail see Methods section). Remaining dataset (80% data of whole dataset) called New training dataset, were used for training, testing and evaluation of our models using five-fold cross validation. Now, each and everything such as parameter optimization, feature selection, model building was done on New training dataset (80% dataset). Final model with optimized parameters and features was used to evaluate performance on validation dataset (this dataset never used in any kind of training process or feature selection). The performance of our models on training and validation is shown in Table 6. As shown in our results on validation dataset are in agreement with training dataset. We also observed that the prediction performance of MACCS 159 keys based model is same for the New training and validation dataset as well as model developed on whole training dataset. However, a slight decrease in MCC value from 0.72 to 0.67 on PCA based model and 0.67 to 0.62 on CfsSubsetEval based model was observed for New Training and validation dataset. This implies that model developed on 159 MACCS keys is suitable for further prediction because the prediction accuracy is highly similar on both New Train and validation dataset. These results suggested that the models developed in this study are not over-optimized.

Quality of written English: Acceptable

Reviewer number 2: Prof Difei Wang (nominated by Dr Yuriy Gusev)

The authors' responses for my questions are acceptable. However, it seems the server still has some problems running examples for virtual screening and design analogs. If possible, it is better to give an estimate of running time. Then the users could decide if they should wait for the results. The output of search database is kind of confusing. The first column gives molecule no. What is this for? Why did the example give the same molecule no. (2) for both ZINC00000053 and CHEMBL505943? It is better to show both database IDs and the corresponding structures.

Response: We are thankful to the reviewer for this valuable suggestion. We have rectified the bug regarding the confusion created by assigning the same molecule number to different compounds. We have also implemented the applet for visualization of chemical structure. Now, the user by clicking on the ID of molecule could visualize the structure of chemical compound. We have also provided the estimated time on the webserver to complete a job. Additionally, an email option has been provided in the webserver. Thus, user will receive a mail after finishing the job.

Quality of written English: Acceptable