VISH-Pred: an ensemble of fine-tuned ESM models for protein ...

Abstract

Peptide- and protein-based therapeutics are becoming a promising treatment regimen for myriad diseases. Toxicity of proteins is the primary hurdle for protein-based therapies. Thus, there is an urgent need for accurate in silico methods for determining toxic proteins to filter the pool of potential candidates. At the same time, it is imperative to precisely identify non-toxic proteins to expand the possibilities for protein-based biologics. To address this challenge, we proposed an ensemble framework, called VISH-Pred, comprising models built by fine-tuning ESM2 transformer models on a large, experimentally validated, curated dataset of protein and peptide toxicities. The primary steps in the VISH-Pred framework are to efficiently estimate protein toxicities taking just the protein sequence as input, employing an under sampling technique to handle the humongous class-imbalance in the data and learning representations from fine-tuned ESM2 protein language models which are then fed to machine learning techniques such as Lightgbm and XGBoost. The VISH-Pred framework is able to correctly identify both peptides/proteins with potential toxicity and non-toxic proteins, achieving a Matthews correlation coefficient of 0.737, 0.716 and 0.322 and F1-score of 0.759, 0.696 and 0.713 on three non-redundant blind tests, respectively, outperforming other methods by over on these quality metrics. Moreover, VISH-Pred achieved the best accuracy and area under receiver operating curve scores on these independent test sets, highlighting the robustness and generalization capability of the framework. By making VISH-Pred available as an easy-to-use web server, we expect it to serve as a valuable asset for future endeavors aimed at discerning the toxicity of peptides and enabling efficient protein-based therapeutics.

Keywords: peptide toxicity, protein toxicity, deep learning, ESM2 models, fine-tuning, ensemble method

Introduction

Proteins are natural biological molecules that are essential for wide variety of fundamental cellular processes and molecular functions, such as enzyme activity [1], energy source [2] or activation of the programmed cell death cascade [3, 4]. However, aberrant protein activity can lead to various diseases, including cancer, diabetes and neurological disorders [5–8]. Because of this, the use of proteins or peptides as therapeutic agents is a promising avenue to fight against myriad diseases. Owing to their cost efficiency, high target specificity and biological activity, they are preferred therapeutics over drugs and antibodies [9]. Moreover, to date, over 80 peptides and proteins have been approved globally for usage in the clinic [10]. Nevertheless, a significant obstacle in the development of protein/peptide-based therapeutics is the potential for adverse effects.

Protein toxicity can have deleterious effects on living organisms. Toxic proteins exist naturally in the plant kingdom as well and are present in abundance in animals such as spiders, snakes, scorpions, jellyfish, etc., and myriad microbes such as bacteria and fungi, enabling their enhanced pathogenecity [11]. In addition, there are synthetic toxins, such as carcinogens and mutagens, with high lethality [12]. Toxic proteins from certain bacteria, such as Listeria monocytogenes, Vibrio cholerae, Citrobacter freundii, can cause deadly diseases such as listeriosis, cholera and meningitis, respectively. The mechanisms of action of the different toxins vary, with snake venom inducing ‘neurotoxicity’ causing neuromusclar damage and paralysis or ‘haemotoxicity’ leading to blood coagulation and cardiac arrest [13, 14], whereas scorpion venom can overstimulate neuronal signaling pathways leading to paralysis [15], and jellyfish venom induces anaphylaxis [16]. Fig. 1 illustrates the impact of toxins from various exposure routes.

Figure 1.

Graphical representation of sources of toxic peptides/proteins in nature and their effects on human health. Figure 1 was created with BioRender.com

Although the conventional approach to determine the toxicity of peptides/proteins through experimental evaluation using toxicity assays [17] is the most reliable, it is often laborious and time-consuming. For this reason, with the advent of novel and powerful computational technologies, it is valuable to design accurate in silico methods for estimating the toxicity of proteins to advance the development of protein-based therapeutics.

The computational methods that have been proposed to date to predict protein toxicity can be broadly classified into three categories: (a) template-based methods, (b) feature-based methods and (c) deep learning-based methods. Template-based techniques utilize protein sequence alignment tools, such as BLAST [18], to measure the sequence similarity between a target protein and a database of template protein sequences whose toxicity is known and to infer the toxicity of the target protein through homology [19, 20]. However, the predictive capacity of these methods hinges on the quality of the template sequence, which is not always available, thereby preventing their widespread application.

Feature-based toxicity predictors such as ClanTox [21], ToxClassifier [22], NNTox [23], ToxinPred [24], ToxinPred2 [25] and ToxinPred3 [26] use a two-step approach. In the first phase, a feature vector representation is devised for individual proteins. This feature vector can represent various properties of the protein, including compositional, evolutionary and physico-chemical features. The compositional features are derived using amino acid compositions such as mono-, di- and tri-petide frequencies as used in BCrystal [27], ProtSol [28] or feature modules from tools such as Pfeature [29] and iFeature [30]. Evolutionary features provide more information than the protein primary sequence [31] and are usually estimated as a position-specific scoring matrix profile, as detailed in [18, 32]. Finally, a variety of numeric physicochemical characteristics such as Cruciani properties, FASGAI vectors, Kidera factors, Aliphatic Index, hydrophobicity, molecular weight, etc., can be extracted using the peptides package (v0.3.1) in Python (v3.7.12) [33]. In the second phase, the extracted features are passed to machine learning models such as support vector machines (SVMs) [34, 35], gradient boosting machine (GBM) [36, 37] and feed-forward neural networks [38].

For example, ToxClassifier generates features for protein representations and then uses traditional machine learning models to identify venom toxins [22]. Similarly, popular tools such as ToxinPred [24] and ToxinPred2 [25] use a comprehensive set of characteristics for peptides in combination with machine learning models such as SVMs and random forests [39] to predict the toxicity of peptides, regardless of the source of the peptide (i.e. animal, plant or bacteria). However, the performance of these methods relies heavily on the extracted set of features, necessitating prior expert knowledge of peptides and proteins to ensure the accuracy of predictions.

With the availability of large-scale biological data of public databases such as Uniprot [40], several data-driven deep learning methods have been designed for protein toxicity prediction [41–45]. While Toxify[41] employed a gated recurrent neural network architecture to identify toxic proteins, ATSE [43] adopted graph neural networks with attention mechanisms [46, 47] to learn efficient and discriminative representations from the evolutionary and structural properties of proteins for toxicity prediction. The authors of ATSE [43] further improved their model using information bottlenecks and a transfer learning framework to devise an enhanced toxicity predictor, namely ToxIBTL [44].

Recently, a transformer-based [48] deep learning technique named CSM-Toxin [45] was developed. CSM-Toxin predicts protein toxicity relying solely on the protein primary sequence and creating the largest curated dataset for toxicity prediction. CSM-Toxin is based on ProteinBERT [49], a deep learning model which follows the same principle as the natural language processing model BERT (Bidirectional Encoder Representations from Transformers) [50]. By treating each amino acid as a word and protein sequences as sentences and using attention mechanisms, BERT can capture complex correlations between even distant residues. While fine-tuning ProteinBERT for protein toxicity prediction, the CSM-Toxin model uses the Global Ontology representation of the protein sequence, i.e. concatenating the outputs of each transformer layer to generate an 8493-dimensional binary vector. One of the drawbacks of CSM-Toxin is that the optimal threshold for identifying toxic proteins is 0.968 due to the drastic class imbalance ( 1:91) in toxic versus non-toxic [45]. This suggests that for a protein to be classified as toxic the predicted probability score for that protein should be , whereas in binary classification the predictions range between [0,1]. Hence, all proteins with toxicity scores are classified as negative by CSM-Toxin, reflecting the huge class imbalance in the training data.

In this work, we build an ensemble of fine-tuned deep learning models using ESM2 [51] as the base model. ESM2 is a state-of-the-art transformer-based protein language model trained on 65 million unique protein sequences [51]. ESM2 has been shown to outperform all tested single-sequence protein language models across a range of structure prediction tasks, enabling atomic resolution structure prediction. While the ESM2 model has been benchmarked for structure prediction, it has not been gauged for protein property prediction and has been shown to not scale for protein function prediction [52]. Moreover, the ESM2 models are available with different architectural configurations, that is, with an increasing number of transformer layers leading to increasing number of model parameters.

Our primary contributions are:

• use the largest dataset for toxicity prediction [45] and handle the humongous class-imbalance by devising a smart under-sampling technique (optimal prediction cutoff = 0.5);

• benchmark different ESM2 model configurations for the task of protein toxicity prediction by fine-tuning and providing insights into size versus performance of the models;

• fine-tuning by updating the weights of all transformer layers of ESM2 model in an end-to-end deep learning framework achieves performance similar to state-of-the-art models;

• utilize a feature-based fine-tuning approach on optimal ESM2 model to generate embedding representation for each protein sequence and pass to state-of-the-art models for tabular data, namely LightGBM [53] and XGBoost [54];

• perform a comprehensive comparison with latest protein toxicity prediction models across myriad quality metrics [accuracy, precision, recall, F1, Matthews correlation coefficient (MCC) etc.] which handle the class imbalance on multiple independent datasets;

• assemble the top-performing fine-tuned models to achieve over 10% improvements in critical quality metrics such as MCC [55] and F1-score [56, 57], given the large class-imbalance in the data;

• provide the VISH-Pred framework as a web-server at http://ec2-35-170-123-194.compute-1.amazonaws.com:7860/ for ease of access of non-experts.

Figure 2 shows a flowchart of the proposed VISH-Pred framework for protein toxicity prediction.

Figure 2.

Representative diagram of the VISH-Pred framework for protein toxicity prediction.

Materials and methods

Data collection and curation

Data preparation

We followed the protocol used by CSM-Toxin [45] to assemble the largest dataset for the task of peptide toxicity prediction. Specifically, the data were collected from UniProt release 2022_04 [58]. Previously reviewed proteins using the query ‘(keyword: KW-0800) AND (reviewed:true)’ were classified as toxic (i.e. positive samples). Those reviewed using the query ‘NOT (keyword:KW-0800) NOT (keyword: KW-0200) AND (reviewed:true)’ were classified as non-toxic and non-allergic proteins (i.e. negative samples). This resulted in a total of 567 390 proteins including 7543 positive and 559 847 negative samples, respectively.

Protein sequences containing nonstandard residues, such as B, J, O, U, X and Z, were discarded, thus reducing the positive set to 7398 and the negative set to 557 354 protein samples, respectively. CD-HIT version 4.8.1 [59] was used with a similarity threshold of 0.7 [60] to remove redundant proteins sets and make the data set unbiased. Thus, proteins with high sequence similarity i.e. point mutations and small truncations were removed and only representative samples were kept to create an unbiased training set. The final real data set covered 2309 positive and 212 541 negative protein samples, respectively. Additionally, the minimum size of a peptide in our dataset is 11 amino acids (AA) and the median size of protein is 302 AAs and the mean protein size was 341 AAs. A distribution of the protein size included in our dataset is depicted as a histogram in Supp. Figure S1.

Data curation

In CSM-Toxin [45], the following steps were undertaken to create an independent test set, a blind test set from the real dataset and the remaining real dataset used for training:

• a small set of 204 toxic and 2337 non-toxic proteins collected after July 2022 and with low sequence similarity to training set was considered as an independent test set;

• a subset comprising 236 positive and 21 294 negative proteins was extracted from the real dataset as a blind set;

• the remaining real dataset (2073 positive and 191 247 negative samples) was considered as a training set and equally split in five parts for cross-validation in [45].

Since the ESM2 model can reliably perform inference only for proteins of size less than 1024 amino acids, we took some additional steps including the following:

• we removed protein sequences of length to have a revised independent test set consisting of 202 toxic and 2160 non-toxic proteins;

• the revised blind set comprised 222 positive and 20 329 negative samples after removing large protein sequences;

• the reduced real dataset had 2015 toxic and 182 561 non-toxic samples after removing proteins with .

• we implemented a simple under-sampling technique to reduce the enormous class imbalance ratio (). Keeping the set of positive samples fixed, we divided the set of negative samples into 10 random groups and fine-tuned ESM2 models for each set comprising 2015 positive samples and 18 256 negative samples, resulting in 10 different models.

• each revised real dataset was stratified into two-parts comprising the training set (80%) and the validation set (20%) while maintaining the class-imbalance;

• each model had an optimal classification cutoff of 0.5 and a reduced class-imbalance ratio () and this setup enabled easy ensemble of fine-tuned ESM2 models.

The data preparation protocol is detailed in Fig. 3. The preprocessing steps used on the training data are consistent with those used for processing the test sets.

Figure 3.

Data preprocessing pipeline. Data were collected from UniProt KB using a keyword search for toxic and non-toxic proteins. Sequences with non-canonical amino acids and high similarity were discarded using CD-HIT. All proteins of length 1024 amino acids were removed from the curated datasets. Finally, a smart under-sampling approach was utilized to reduce the class-imbalance and divide the real dataset into 10 subsets.

Independent bacterial dataset

An independent dataset comprising peptides with antibacterial activity as well as toxic bacterial peptides was obtained from [61]. This dataset comprised 2170 peptides, all below the sequence similarity threshold of 0.7 with real dataset using CD-HIT and obtained from the public source named Database of Antimicrobial Activity and Structure of Peptides (DBAASP) [62]. Each peptide sequence varied in length between 4 and 119 AAs and were classified into two groups, i.e. the non-toxic class with high antibacterial activity and low cytotoxicity and the toxic class with low antibacterial activity and high cytotoxicity. The cutoff values to annotate a peptide as toxic was minimum inhibitory concentration M, or cytotoxic concentration at () M, or hemolytic concentration at () M. There were a total of 1071 non-toxic peptides and 1099 toxic peptides in this independent bacterial dataset.

Methods overview

Protein toxicity prediction can be computationally modeled as a classification task. We learn a mapping function that takes as input a protein representation, say , and outputs the predicted toxicity score , where . If is the model-specific loss function, then the classification task reduces to estimating the parameters which minimizes , where is the ground truth label and takes the value 0 (for non-toxic proteins) or 1 (for toxic proteins).

In this article, the mapping function is based on the ESM2 model [51]. ESM2 is a transformer-based protein language model trained on 65 million unique protein sequences using a masked language modeling objective [63]. The ESM2 model is trained to predict the identity of randomly selected amino acids in a protein sequence by observing their context w.r.t. rest of the sequence. This enables the model to learn dependencies between the amino acids. While the training objective of the ESM2 model is relatively simple and unsupervised, by performing well on this task over millions of diverse proteins, the model can internalize sequence patterns across evolution. As a result, the model can efficiently capture structure as it can be linked to the sequence patterns. In addition, the ESM2 models were trained with different configurations as depicted in Supplementary Table S1. Increasing the size of the models lead to large improvements in quality of protein structure [51].

In our work, we fine-tune the ESM2 model transformer with classification head [51] for the binary classification, i.e. toxic versus non-toxic, based on the sequence. Since the dataset used for fine-tuning the ESM2 models are highly imbalanced, we used the weighted binary cross-entropy loss as the loss function () which can formally described as follows:

(1)

where is the number of proteins, and are the weights for the classes 1 and 0 and are chosen to represent the imbalance in the dataset, i.e. and . is the probability of class 1 and is the probability of class 0. By minimizing the loss function , we can fine-tune the models to accurately estimate the toxicity of proteins.

Fine-tuning ESM2 models

We downloaded and fine-tuned the ESM2 models of varying size from the HuggingFace library [64] in Python and followed a protocol. Since we divided our reduced real dataset into 10 parts, we fine-tuned one ESM2 model for each part while keeping configuration fixed (i.e. defined number of transformer layers and learnable parameters). The output of ESM2 model (i.e. the last transformer layer) is a local representation of the protein sequence and is connected with a trainable classification head comprising feed forward layers followed by a softmax layer to categorize proteins into toxic or non-toxic classes. This is shown in Fig. 4A. Each ESM2 transformer model, along with the classification head, is fine-tuned in the corresponding training set and evaluated in the validation set using the Adam optimizer [65].

Figure 4.

VISH-Pred model architecture. (A) 10 ESM2 models handle the humongous class imbalance and are fine-tuned for the protein toxicity prediction task. All the parameters for the boxes highlighted in ‘green’ are fine-tuned. (B) The fine-tuned ESM2 transformer model with the best performance in the validation set generated the local feature representations for each of the 10 parts of the reduced real data set. Each part (1 out of 10) is utilized to build an optimal LightGBM classifier via a 5-fold cross-validation. (C) The same local feature representations for each of the revised real dataset is used to build an optimal XGBoost classifier (through 5-fold cross-validation). The outputs of the 10 fine-tuned models, the 10 LightGBM classifiers and the 10 XGBoost classifiers are passed to the ensemble classifier (VISH-Pred) to have the final output score ) for a given protein .

The Adam optimizer depends on several parameters including learning rate, batch size, maximum number of epochs and early stopping criterion as described in [57, 66]. We fine-tuned each model for a maximum of 10 epochs with a learning rate of 1e-4 and weight decay of 0.01. A low learning rate and weight decay are required to prevent the appearance of large absolute values of the gradients that can significantly change the pretrained model weights. We used a single Nvidia RTX A6000 GPU with 48GB RAM for the fine-tuning process. We therefore used only those ESM2 models that could be loaded on a single GPU for fine-tuning. In addition, as the size of the ESM2 model increases, the number of samples that can be passed to the model in a given batch decreases. To have a fair comparison between ESM2 models of varying configurations, we limited the batch size to 4 and introduced a gradient accumulation step [64]. Gradient accumulation is a way to virtually increase the batch size during the fine-tuning process and is useful when the available GPU memory is insufficient to accommodate the desired batch size. In gradient accumulation, gradients are computed for smaller batches and accumulated (summed or averaged) over multiple iterations instead of updating the model weights after every batch. We fixed the gradient accumulation step to 4, thus virtually increasing the batch size to 16. Additionally, we set the early stopping criterion to 5, i.e. if the model performance on the validation set does not improve for 5 consecutive epochs, then we stop training the model further to avoid over-fitting. As our dataset is heavily imbalanced, we implemented our own weighted loss trainer to inform the model to give more weights to the toxic proteins (class 1).

Once we fine-tuned each ESM2 transformer model along with the classification head (1 out of 10 parts), we concatenated the outputs of the 10 models for the ensemble classifier. We also identified the ESM2 transformer model classifier (1 out of 10) with the best performance on validation set as depicted in Fig. 4A. The output feature representations from the corresponding fine-tuned ESM2 transformer model are similarly used by other state-of-the-art machine learning classifiers, such as LightGBM and XGBoost. Fig. 4A highlights the fine-tuning process of the ESM2 transformer model along with the classification head on each revised real dataset. Their generated outputs are concatenated and used in the ensemble classifier.

Traditional machine learning models

It has been previously shown that nonlinear ML techniques such as XGBoost and LightGBM can be used efficiently for a variety of bioinformatics problems [6, 27, 28, 37, 56, 67–69]. Here we too used such state-of-the-art nonlinear ML methods for tabular data, XGBoost [27, 54] and LightGBM [53], as mapping function . Both XGBoost and LightGBM are based on GBMs [36]. GBM belongs to a family of predictive methods that uses an iterative strategy, that is, the learning framework will consecutively fit new models to have an accurate estimate of the response variable after each iteration. The advantage of the boosting procedure is that it works on decreasing the bias in the model, without increasing the variance. A more scalable and accurate version of GBM is XGBoost [54] and light-weight version of GBM is LightGBM [53]. These methods use a scalable end-to-end tree-boosting system with a weighted quantile sketch for approximate tree learning. Both XGBoost and LightGBM can scale for a large number of samples using very little computational resources.

Here, the input to these models are the output embeddings from the last transformer layer of the best-performing fine-tuned ESM2 model resulting in the feature representation vector, , for the protein . Thus, our predicted toxicity for a given protein can be formally written as . We used the ‘xgboost’ package (v1.7.6) and ‘lightgbm’ package (v4.0.0) available in Python for building XGBoost and LightGBM models. For each revised real dataset, we build one optimal XGBoost (see Fig. 4C) and one optimal LightGBM model (see Fig. 4B), respectively, resulting in 10 optimal models for the 10 parts. We perform hyperparameter optimization using 5-fold stratified cross-validation. To do cross-validation, we shuffled the rows of each revised real dataset and then randomly split the data into five parts, using a combination of four parts as training set and one part as validation set (while maintaining the class-imbalance) to identify the optimal set of hyperparameters. This process is repeated five times and the hyperparameters with best average performance are then selected as optimal hyperparameters. These hyperparameters are used to build the final model on the entire revised real dataset.

VISH-Pred: ensemble classifier

Ensemble methods use multiple learning algorithms to obtain better predictive performance than any individual component [70] for a given supervised task. For instance, in [71], the authors demonstrated that, for identifying drugs that can be repurposed against COVID-19, an ensemble of the results from different methodologies can provide better performance than individual models. In a similar vein, we take a consensus, i.e. the average () of the predicted toxicity scores from the top-performing fine-tuned ESM2 transformer model with classification head, XGBoost and LightGBM models on independent test sets as the output of the ensemble classifier (). We claim that since our models are based on different configurations (ESM2 model) and learning algorithms (tree-based versus end-to-end deep learning), it is essential to take an ensemble of the top predictive models to attain optimal performance as illustrated in Table 1. Supp. Figure S2 highlights detailed architecture of the proposed VISH-Pred framework.

Evaluation metrics

Following [56, 57], the performance of proposed VISH-Pred ensemble classifier was compared with various other in silico protein toxicity predictors using quality metrics such as accuracy, area under receiver operating curve (AUC), and MCC. We assessed several other evaluation metrics, based on TP, TN, false positives (FP) and false negative (FN). TP represents the set of proteins which are toxic (true label is 1) and are correctly identified by a given method as toxic, i.e. . Similarly, TN represents the set of proteins which are non-toxic (true label is 0) and are correctly identified by a given method as non-toxic (). Based on the same principle, the score distribution for the FP set represents the score distribution for all proteins whose true label is 0 but are incorrectly identified as toxic. The score distribution for the FN set represents the score distribution for all proteins whose true label is 1 but are incorrectly identified as non-toxic. The metrics used for evaluation include

Finally, each of previous toxicity predictor was downloaded from their respective github repository (except CSM-Toxin) and was utilized in an inference phase to predict toxicity for test proteins/peptides with default parameter settings. For CSM-Toxin, the protein sequences were passed to their web-server as a FASTA file and the quality metrics were applied on the obtained results accordingly.

Docking protocol for case study

The goal of the molecular docking investigation was to find a common ligand (small molecule) with high binding affinity against two hazardous peptides from Escherichia coli and Klebsiella pneumoniae, which were correctly identified by the VISH-Pred model as toxic proteins.

Due to the unavailability of the 3D structures of these target peptides, homology modeling of target sequences was performed using AlphaFold [72]. Although AlphaFold uses deep learning to predict protein structures, it does so within the context of homology modeling. Traditionally, homology modeling methods such as SwissModel [73] and ModWeb [74] involve comparing similar protein structures to predict the structure of a target protein. AlphaFold makes this prediction by utilizing evolutionary connections relevant to the protein sequence, similar to such standard homology modeling approaches, using a cutting-edge deep learning model. AlphaFold model provides high predicted local Distance Difference Test (plDDT) scores for the target structures [75]. High plDDT scores (e.g. > 80) indicate high confidence in the structure of the residue, and low plDDT scores (e.g. < 50) indicate that the residues are in intrinsically disordered protein regions [72]. Supp. Figures S3a and S3b highlight the target structures along with their corresponding plDDT scores. To restore equilibrium in the modeled structures, a molecular dynamics simulations was run for 100 ns.

The ChEMBL database was selected for the molecular docking study [76] comprising 1000 compounds with known antibacterial activity [77]. The compounds chosen from the ChEMBL database include a diverse range of molecules, including those that have been shown to bind to toxic proteins, as well as medications that target disorders caused by toxic proteins/peptides. We have ensured that the selection criteria are consistent with our research goals of identifying possible interactions between relevant chemicals and biological targets. Finally, molecular docking was employed using the Maestro module of Schrödinger software (Release 2023-2) using pre-defined parameters for protein and compound preparation.

Next, the homology modeled structures were then processed through the Schrödinger’s Glide (grid-based ligand docking with extra precision) protein preparation wizard, which included pre-processing steps such as adding hydrogens, removing water and optimizations carried out using a refinement technique that aids the OPLS 2005 force field [78, 79]. Furthermore, the Ligprep suite of the Maestro module was utilized for compound minimization (we applied a flexible docking approach because of the small size of the peptides to obtain the docking score and inter-molecule interactions). Finally, the binding pocket dimension were set with grid box size and the docking experiment was performed with the Schrödinger molecular modeling package’s Glide program.

Experimental results

Benchmarking on validation sets

We benchmarked the performance of the fine-tuned ESM2 transformer models with classification head and different configurations (in terms of transformer layers and parameter size) as reported in Supp. Table S2. From Supp. Table S2, it can be seen that in the validation sets for the 10 parts of the reduced real dataset, the performance of the ESM2 models increased as the size of the model increased. The mean standard deviations of the F1 and MCC scores for ESM2: T6-12M, ESM2: T12-35M, ESM2: T30-150M and ESM2: T33-650M models were 0.882 0.015, 0.894 0.011, 0.908 0.014, 0.909 0.012 and 0.869 0.017, 0.882 0.027, 0.898 0.013, 0.899 0.022, respectively, in the validation sets as observed in Supp. Table S2 and Supp. Figures S4a–d.

However, we observed that feature-based models such as XGBoost (XGB) and LightGBM (LGBM) models built on top of features generated by best fine-tuned ESM2 model (of a particular configuration) had better performance w.r.t. all evaluation metrics as illustrated in Supp. Figures S4e–l. The LGBM (ESM2: T6-8M), LGBM (ESM2: T12-35M), LGBM (ESM2: T30-150M) and LGBM (ESM2: T33-650M) models achieved F1 scores of 0.942 0.015, 0.945 0.013, 0.947 0.009, 0.952 0.009 and MCC scores of 0.936 0.017, 0.939 0.014, 0.942 0.010, 0.947 0.010, respectively, on the validation sets as depicted in Supp. Table S2 and Figures S4e–h. Similarly, XGB (ESM2: T6-8M), XGB (ESM2: T12-35M), XGB (ESM2: T30-150M) and XGB (ESM2: T33-650M) models attained F1 scores of 0.942 0.015, 0.943 0.014, 0.948 0.010, 0.954 0.008 and MCC scores of 0.936 0.017, 0.937 0.015, 0.943 0.011, 0.949 0.009, respectively, on the validation sets as highlighted in Supp. Table S2 and Figures S4i–l. We observed from Supp. Table S2 that XGBoost and LightGBM models could handle class imbalance better since these models directly optimized the F1 scores during cross-validation compared with the fine-tuned ESM2 transformer models with classification head which optimized the weighted binary cross-entropy loss (). Finally, the ESM2: T30-150M and ESM2: T33-650M along with their feature-based counterparts, XGB (ESM2: T30-150M), XGB (ESM2: T33-650M), LGBM (T30-150M) and LGBM (T33-650M) had better performance than the small sized ESM2 models (see Supp. Table 2). Hence, we took the ensemble of these models to build our final ensemble classifier, VISH-Pred.

We also benchmarked the run time complexity of fine-tuning the ESM2 models as indicated in Supp. Table S3 and Supp. Figure S5a. All the ESM2 models ran up to the maximum setting of 10 epochs as illustrated in Supp. Figure S5b. The run time increased quadratically as the size of the ESM2 model increased from 8 million parameters ( s) to 650 million parameters ( s). Here represents the mean run time in seconds, i.e. the average run time for fine-tuning each ESM2 model on a revised real dataset. Additionally, the training loss decreased with the number of epochs, while the validation loss decreased initially and increased with the number of epochs, as highlighted in Supp. Figure S5b. However, the validation MCC which is used as the early stopping criterion was maximum around the or epoch for most of the fine-tuned ESM2 models and as a result majority of the models ran up to the maximum 10 epochs as depicted in Supp Figure S5b.

Performance on small test set

We compared the predictive performance of state-of-the-art toxicity predictors such as ToxClassifier, ToxITBL, Toxify, ToxinPred2, ToxinPred3 (ET), ToxinPred3 (Hybrid) with fine-tuned ESM2 models and our ensemble classifier, VISH-Pred, across different quality metrics on the small test set as depicted in Table 1 and Fig. 5(A). The class-imbalance ratio in this dataset was 1:10. We observed that the VISH-Pred model achieved the best performance with respect to metrics (w.r.t.) such as F1 (0.759), ACC (0.956) and MCC (0.737) and second best performance for AUC (0.891) as illustrated in Table 1. The performance of VISH-Pred model was superior than other state-of-the-art toxicity predictors with over improvement w.r.t. F1 and MCC scores over the next best peptide toxicity predictor (CSM-Toxin).

Table 1.

Comparison of performance of fine-tuned ESM2 models, ensemble classifier (VISH-Pred) with other state-of-the-art in silico peptide toxicity predictors on the small independent test set. Here ‘’ represents the best model and ‘’ represents the second best model

Figure 5.

Comprehensive comparison and benchmarking of the fine-tuned ESM2 models and the ensemble classifier (VISH-Pred) against state-of-the-art methods such as Toxify, ToxClassifier, ToxIBTL, CSM-Toxin, ToxinPred2, ToxinPred3 on three independent test sets. The models are evaluated using various evaluation metrics, in particular, metrics such as MCC and F1-score to handle the class imbalance in the test sets.

The fine-tuned ESM2 models (ESM2: T30-150M and ESM2: T33-650M) achieved the best AUC (0.894) and second best AUC (0.891), respectively, while the feature-based LGBM (ESM2: T12-35M) achieved the joint best accuracy (0.956) along with the VISH-Pred model as highlighted in Fig. 5(A). Similarly, the ESM2: T30-150M achieved the second best recall score (0.824) following ToxinPred2 (0.842) and the XGB (ESM2: T6-8M) achieved the best precision score (0.724) on the small test set. The majority of the fine-tuned ESM2 models already attained performance better than state-of-the-art methods such as CSM-Toxin, ToxinPred2 and ToxinPred3 for several quality metrics.

Performance on blind test set

We compared the performance of proposed methods with other protein toxicity predictors on the big blind test set where the class imbalance ratio was 1:91. As a result, the predictive performance of all methods decreased, in particular, for the F1 and MCC quality metrics, which handle the class-imbalance in the data as depicted in Table 2. VISH-Pred model accomplished the best scores for MCC (0.716), AUC (0.991) metrics and second best performance for F1 (0.696) and ACC (0.991) metrics as showcased in Fig. 5(B) and Table 2. Interestingly, LGBM (ESM2: T33-650M) and XGB (ESM2: T33-650M) models procured optimal performance w.r.t. F1, ACC, MCC and precision metrics. Furthermore, the ToxClassifier model achieved the best accuracy (0.993) and best precision (0.639) at the expense of much lower recall (0.703) compared with the fine-tuned ESM2 models. Similarly, ToxinPred2 secured the best recall (0.991) at the expense of extremely low precision (0.132). Although the VISH-Pred model cannot outperform individual peptide toxicity predictors on precision and recall, it achieved an increase of over in performance for relevant quality metrics (F1 and MCC) over the next best model (ToxClassifier) given the humongous class-imbalance in the blind test set.

Table 2.

Comparison of performance of fine-tuned ESM2 models, ensemble classifier (VISH-Pred) with other state-of-the-art in silico peptide toxicity predictors on the big blind test set. Here ‘’ represents the best model and ‘’ represents the second best model

Performance on bacterial test set

Finally, we compared our proposed framework with other state-of-the-art protein toxicity predictors in the independent bacterial test set. The bacterial test set was balanced in terms of number of toxic and non-toxic bacterial peptides. However, the sequence similarity with our training set was low and the lengths of bacterial peptides were 120 amino acids. This differs significantly from the length distribution of the reduced real dataset utilized for fine-tuning the ESM2 models as highlighted in Supp. Figure S1. This resulted in a significant drop in performance of all methods for various quality metrics including accuracy (ACC), AUC and MCC.

The VISH-Pred model still achieved the best performance w.r.t. different quality metrics including F1 (0.713), AUC (0.644), ACC (0.647), MCC (0.322) and second best performance for precision (0.606). ToxITBL attained the best precision (0.65) at the expense of very low recall (0.264) and ToxinPred3 (ET) had the best recall (0.905) but relatively low precision (0.524). In line with our previous observation, the VISH-Pred model achieved over performance for F1 and MCC quality metrics than the next best predictor, i.e. ToxinPred3 (ET) and ToxITBL respectively. Additionally, on this difficult independent test set, VISH-Pred impressively accomplished over gain in performance for AUC and ACC quality metrics than the competing protein toxicity predictors.

Case study on bacterial test set

We undertook a case study of the two most toxic peptides from the bacterial test set and were accurately identified by the VISH-Pred model, one from Escherichia coli and one from Klebsiella pneumoniae, respectively, as indicated in Supp. Table S4. We wanted to identify whether a known small molecule could bind and inhibit the toxic peptides. To answer this, we performed a docking study to identify a common ligand with high binding affinity against both toxic peptides.

We first showed that the 3D structures of the peptides had more than predicted local Distance Difference Test (plDDT) scores via AlphaFold, as highlighted in Supp. Figures S2a and S2b. The plDDT scores can take values in the range of . High plDDT scores (e.g. ) indicate high confidence in the residue structure, and low plDDT scores (e.g. ) indicate lower confidence in the estimated structure. The homology modeled structures were docked with CheMBL compounds and top-10 hits were identified based on docking scores (see Supp. Tables S5 and S6).

Among the top-hits, Carbovir Triphosphate was pinpointed as a potential binder (inhibitor) with both the peptides. The docking of Carbovir Triphosphate and E. coli peptide formed one hydrogen bond (Lys 18) and two salt bridges (Lys 18 and Lys 22) with the binding energy of -5.33 Kcal / mol as shown in Fig. 6(A). Similarly, the peptide obtained from Klebsiella pneumoniae and Carbovir Triphosphate docked with a binding energy of -5.78 Kcal/mol forming four hydrogen bonds (Lys 31, Asn 35 and Thr 65) as illustrated in Fig. 6(B). Based on both docking incidents (see Supp. Tables S5 and S6), we can conclude that the same compound can potentially act as an inhibitor to target highly toxic peptides from two different model organisms.

Figure 6.

Carbovir Triphosphate binding with residues of toxic peptides from E. coli and K. pneumoniae, respectively.

Here we recognize that binding affinity is one of the steps in antidote preparation. A good antidotal action is dependent on factors beyond binding affinity. Although docking studies can give significant insight into possible binding interactions, the development of an effective antidote requires a thorough examination across several parameters. These include binding selectivity, functional effects on the target molecule, pharmacokinetic characteristics, safety profile, dose–response relationship, delivery method and clinical validation via preclinical and clinical trials. As a result, while our docking experiments give information on possible interactions between small molecules and hazardous peptides or proteins, the eventual evaluation of these compounds into effective antidotes requires extensive research across numerous domains to ensure therapeutic efficacy and safety.

Finally, we provide a small subset of 36 bacterial proteins as an example file on our web-server and it takes approximately min for the VISH-Pred framework to provide the results.

Discussion

A significant obstacle in protein or peptide therapeutics is the hazard of protein toxicity. It is imperative to evaluate the toxicity of proteins where conventional experimental techniques are expensive and time-consuming. Hence, precise in silico models that predict the toxicity of proteins with high efficacy and high performance are highly desirable and the motivation behind the proposed VISH-Pred framework. The word ‘VISH’ in Sanskrit refers to venom/poison, and hence we named our framework as VISH-Pred.

Using the largest curated dataset of 2015 toxic and 182 561 non-toxic proteins [45], we performed a comprehensive comparison of various classes of in silico protein toxicity predictors including feature-based methods such as ToxClassifier, ToxinPred2 and ToxinPred3 as well as end-to-end deep learning approaches such as Toxify, ToxIBTL and CSM-Toxin. We observed low predictive performance for these methods particularly on quality metrics such as F1-score and MCC on two independent datasets, a small test set comprising 202 toxic and 2160 non-toxic proteins and a bigger test set consisting of 222 toxic and 20 329 non-toxic proteins, respectively. This was indicative of the poor generalization and handling of class imbalance by the prior predictive tools.

Employing the ESM2 model as the base transformer model [51] and the simplest representation of protein i.e. amino acid sequence as input, we designed VISH-Pred, a novel framework for protein toxicity estimation. By just taking the protein sequence as input and no additional computationally expensive post-processing step, such as BLAST or motif scan as implemented by ToxinPred2 and ToxinPred3, VISH-Pred is an efficient high-throughput protein toxicity screening tool. We fine-tuned ESM2 transformer models of various architectural configurations (see Supp. Tables S1 and S2 and Supp. Figure S4) and identified the optimal fine-tuned ESM2 model for the task of predicting protein toxicity. The output embedding representation of these models was then passed to tree-based machine learning methods such as LightGBM [53] and XGboost [54] classifiers. By taking an ensemble of fine-tuned ESM2 models and corresponding feature-based XGBoost and LightGBM models, we create the final VISH-Pred framework. The VISH-Pred framework was able to accurately and robustly identify both toxic and non-toxic proteins, outperforming a priori approaches by at least 10 across three independent test sets for different quality metrics (specifically the F1 and MCC quality metrics).

An important step undertaken in the VISH-Pred framework was to handle the humongous class imbalance in the reduced real dataset. By fixing the toxic proteins as one part and dividing the non-toxic proteins into 10 parts, we reduced the class-imbalance ratio from 1:91 to 1:9, allowing the usage of the cutoff value of 0.5 as the optimal threshold for the protein toxicity classification. Furthermore, by implementing custom weighted trainer for the weighted loss function (i.e. ) in the ESM2 models with classification head and providing the inverted imbalance ratio as the weights to ‘scale_pos_weight’ parameter in the LightGBM and XGBoost models, we observed that these models already achieved performance better than the state-of-the-art for F1, AUC, ACC and MCC metrics as depicted in Tables 1 and 2. Additionally, the XGBoost and LightGBM models achieved better F1 and MCC scores than the fine-tuned ESM2 models (see Tables 1 and 2) as they directly optimized for the F1-score during cross-validation.

By outperforming other models on F1 and MCC metrics (see Tables 1, 2 and 3), the VISH-Pred model can not only accurately identify toxic proteins but has fewer falsely classified toxic proteins (FP) and fewer incorrectly classified non-toxic proteins (FN) simultaneously. This is different from other classifiers such as ToxClassifier and ToxIBTL (see Tables 2 and 3) which can accurately expand the pool of non-toxic proteins (high precision) but at the expense of misclassifying several toxic proteins as non-toxic (FN) proteins. This can be hazardous when designing protein-based therapeutics. Similarly, methods such as ToxinPred2 and ToxinPred3 (see Tables 1, 2 and 3) focus on not missing out on any toxic protein (high recall) but at the expense of misclassifying several non-toxic candidates as toxic. This is also detrimental for the protein-based biologic development as potentially feasible protein candidates rue being missed. The advantage of the proposed VISH-Pred framework is that it attains both high precision and high recall simultaneously as depicted from our results.

Table 3.

Comparison of performance of VISH-Pred with other state-of-the-art in silico peptide toxicity predictors on the bacterial test set. Here ‘’ represents the best model and ‘’ represents the second best model

In a difficult independent bacterial test set comprising predominantly small peptides, most of the previous methods achieved poor predictive performance (MCC < 0.1) as shown in Table 3. VISH-Pred was the only model that achieved a reasonable MCC score (MCC = 0.322) and outperformed other methods by more than for quality metrics such as F1, AUC, ACC and MCC metrics (see Fig. 5C). Additionally through a docking-based study, we illustrated that a common ligand (Carbovir Triphosphate) can render two most toxic peptides, (one from E. coli and one from K. pneumoniae) identified by the VISH-Pred model from the bacterial test set, to be ineffective.

In conclusion, we propose a novel ensemble framework based on fine-tuned ESM2 protein language models, namely VISH-Pred, for the task of protein toxicity prediction. VISH-Pred framework takes just the primary sequence representation of proteins (peptides) as input and overcomes limitations such as a two-stage classifier with a separate feature selection step and handling of the humongous class imbalance in the data. VISH-Pred significantly outperforms all sequence-based protein toxicity predictors with respect to various evaluation metrics such as ACC, MCC, AUC and F1-score across different independent test sets. To facilitate ease of usage, our model is available through an easy-to-use web interface. We expect that VISH-Pred will serve as a valuable asset for upcoming endeavors aimed at discerning the toxicity of proteins, enabling efficient protein-based therapeutics.

Key Points

• Ensemble of fine-tuned ESM2 models (VISH-Pred) outperforms existing state-of-the-art protein toxicity predictors like CSM-Toxin, ToxinPred2, ToxinPred3, ToxClassifier, ToxIBTL, Toxify, etc. on three independent test sets.

• VISH-Pred efficiently handles humongous class-imbalance in training set (ratio of 1:91) for toxic versus non-toxic peptides using an under-sampling approach and ensembles fine-tuned models to mitigate the risk of majority class prediction.

• VISH-Pred achieves over 10% increase in performance for key metrics, F1-score and MCC, which are imperative for screening candidates for protein-based therapeutics.

• Benchmarking different ESM2 language model configurations provided insights into size versus performance trade-offs.

• VISH-Pred is available as a user-friendly web service for the ease of usage of nonexperts.

Supplementary Material

VISH_Pred_Supplementary_bbae270

Acknowledgements

No funding is applicable.

Author Biographies

Raghvendra Mall is a Director at Technology Innovation Institute. His expertise lies in machine learning and computational biology. His research focuses on inferring gene regulatory networks, differential network analysis of biological networks and protein property prediction using deep learning.

Ankita Singh is a Senior Scientist at the Technology Innovation Institute (TII) with a background in biophysics and computational biology. Her research is focused on developing new computational methods and software tools and biological databases to help answer fundamental biological questions.

Chirag N. Patel is a Senior Scientist at the Technology Innovation Institute (TII). His areas of interest include computational biology investigations for biomedical treatments to enhance pharmacological efficacy, including molecular modeling and applied machine learning.

Gregory Guirimand is a Lead Researcher at Technology Innovation Institute. His expertise lies in synthetic biology and metabolic engineering of microorganisms for the bioproduction of valuable compounds.

Filippo Castiglione is the Executive Director of Bioinformatics at Technology Innovation Institute. He has expertise in computational biology, machine learning, mathematical modeling and complex systems with a focus on the immune system.

Contributor Information

Raghvendra Mall, Biotechnology Research Center, Technology Innovation Institute, P.O. Box 9639, Abu Dhabi, United Arab Emirates.

Ankita Singh, Biotechnology Research Center, Technology Innovation Institute, P.O. Box 9639, Abu Dhabi, United Arab Emirates.

Chirag N Patel, Biotechnology Research Center, Technology Innovation Institute, P.O. Box 9639, Abu Dhabi, United Arab Emirates.

Gregory Guirimand, Biotechnology Research Center, Technology Innovation Institute, P.O. Box 9639, Abu Dhabi, United Arab Emirates; Graduate School of Science, Technology and Innovation, Kobe University, 1-1 Rokkodai-cho, Nada-ku, Kobe, 657-8501, Japan.

Filippo Castiglione, Biotechnology Research Center, Technology Innovation Institute, P.O. Box 9639, Abu Dhabi, United Arab Emirates; Institute for Applied Computing, National Research Council of Italy, Via dei Taurini, 19, 00185, Rome, Italy.

Author contributions statement

R.M., A.S. and F.C. conceived the study. R.M., A.S. and C.P. collected and curated the data. R.M. built the models. R.M. and C.P. conducted the experiment(s). R.M., A.S., C.P. and G.G. analyzed the results. All authors contributed in writing, reviewing and revising the manuscript.

Data availability

All the data required to train the model are available via https://bitbucket.org/ascherslab/csm-toxin/src/master/data_processing/ (accessed on 17 January 2023) and VISH-Pred model is available as a web-server for testing at: http://ec2-35-170-123-194.compute-1.amazonaws.com:7860/.

References

• 1. Theodore Lewis and William L Stone . Biochemistry, proteins enzymes. StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing, 2024. [PubMed]
• 2. Hoffman JR, Falvo MJ. Protein–which is best?  Journal of sports science & medicine  2004;3:118–30. [PMC free article] [PubMed] [Google Scholar]
• 3. Karki R, Lee SJ, Mall R. et al.  Zbp1-dependent inflammatory cell death, panoptosis, and cytokine storm disrupt ifn therapeutic efficacy during coronavirus infection. Sci Immunol  2022;7:eabo6294. 10.1126/sciimmunol.abo6294. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 4. Sundaram B, Pandian N, Mall R. et al.  Nlrp12-panoptosome activates panoptosis and pathology in response to heme and pamps. Cell  2023;186:2783–2801.e20. 10.1016/j.cell.2023.05.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 5. Mall R, Bynigeri RR, Rajendra Karki RK. et al.  Pancancer transcriptomic profiling identifies key panoptosis markers as therapeutic targets for oncology. NAR cancer  2022;4:zcac033. 10.1093/narcan/zcac033. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 6. Ullah E, Mall R, Rawi R. et al.  Harnessing Qatar biobank to understand type 2 diabetes and obesity in adult qataris from the first Qatar biobank project. J Transl Med  2018;16:1–10. 10.1186/s12967-018-1472-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 7. Frattini V, Pagnotta SM, Tala JJ. et al.  A metabolic function of fgfr3-tacc3 gene fusions in cancer. Nature  2018;553:222–7. 10.1038/nature25171. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 8. Ghosh A, Mizuno K, Tiwari SS. et al.  Alzheimer’s disease-related dysregulation of mrna translation causes key pathological features with ageing. Transl Psychiatry  2020;10:192. 10.1038/s41398-020-00882-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 9. Bruno BJ, Miller GD, Lim CS. Basics and recent advances in peptide and protein drug delivery. Ther Deliv  2013;4:1443–67. 10.4155/tde.13.104. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 10. Usmani SS, Bedi G, Samuel JS. et al.  Thpdb: database of fda-approved peptide and protein therapeutics. PloS One  2017;12:e0181748. 10.1371/journal.pone.0181748. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 11. Clark GC, Casewell NR, Elliott CT. et al.  Friends or foes? Emerging impacts of biological toxins. Trends Biochem Sci  2019;44:365–79. 10.1016/j.tibs.2018.12.004. [DOI] [PubMed] [Google Scholar]
• 12. Ames BN, Profet M, Gold LS. Nature’s chemicals and synthetic chemicals: comparative toxicology. Proc Natl Acad Sci  1990;87:7782–6. 10.1073/pnas.87.19.7782. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 13. Casewell NR, Jackson TNW, Laustsen AH. et al.  Causes and consequences of snake venom variation. Trends Pharmacol Sci  2020;41:570–81. 10.1016/j.tips.2020.05.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 14. Slagboom J, Kool J, Harrison RA. et al.  Haemotoxic snake venoms: their functional activity, impact on snakebite victims and pharmaceutical promise. Br J Haematol  2017;177:947–59. 10.1111/bjh.14591. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 15. Petricevich VL. Scorpion venom and the inflammatory response. Mediators Inflamm  2010;2010:1–16. 10.1155/2010/903295. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 16. Tibballs J, Yanagihara AA, Turner HC. et al.  Immunological and toxinological responses to jellyfish stings. Inflammation & Allergy-Drug Targets (Formerly Current Drug Targets-Inflammation & Allergy)(Discontinued)  2011;10:438–46. 10.2174/187152811797200650. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 17. Duracova M, Klimentova J, Fucikova A. et al.  Proteomic methods of detection and quantification of protein toxins. Toxins  2018;10:99. 10.3390/toxins10030099. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 18. Altschul SF, Madden TL, Schäffer AA. et al.  Gapped blast and psi-blast: a new generation of protein database search programs. Nucleic Acids Res  1997;25:3389–402. 10.1093/nar/25.17.3389. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 19. Negi SS, Schein CH, Ladics GS. et al.  Functional classification of protein toxins as a basis for bioinformatic screening. Sci Rep  2017;7:13940. 10.1038/s41598-017-13957-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 20. Litfin T, Yang Y, Zhou Y. Spot-peptide: template-based prediction of peptide-binding proteins and peptide-binding sites. J Chem Inf Model  2019;59:924–30. 10.1021/acs.jcim.8b00777. [DOI] [PubMed] [Google Scholar]
• 21. Naamati G, Askenazi M, Linial M. Clantox: a classifier of short animal toxins. Nucleic Acids Res  2009;37:W363–8. 10.1093/nar/gkp299. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 22. Gacesa R, Barlow DJ, Long PF. Machine learning can differentiate venom toxins from other proteins having non-toxic physiological functions. PeerJ computer Science  2016;2:e90. 10.7717/peerj-cs.90. [DOI] [Google Scholar]
• 23. Jain A, Kihara D. Nntox: gene ontology-based protein toxicity prediction using neural network. Sci Rep  2019;9:17923. 10.1038/s41598-019-54405-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 24. Gupta S, Kapoor P, Chaudhary K. et al.  Open source drug discovery consortium, and Gajendra PS Raghava. In silico approach for predicting toxicity of peptides and proteins. PloS One  2013;8:e73957. 10.1371/journal.pone.0073957. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 25. Sharma N, Naorem LD, Jain S. et al.  Toxinpred2: an improved method for predicting toxicity of proteins. Brief Bioinform  2022;23:bbac174. [DOI] [PubMed] [Google Scholar]
• 26. Rathore AS, Arora A, Choudhury SPS. et al.  Toxinpred 3.0: An improved method for predicting the toxicity of peptides  bioRxiv. 2023:2023–08.
• 27. Elbasir A, Mall R, Kunji K. et al.  Bcrystal: an interpretable sequence-based protein crystallization predictor. Bioinformatics  2020;36:1429–38. 10.1093/bioinformatics/btz762. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 28. Mall R. Solxplain: an explainable sequence-based protein solubility predictor  BioRxiv. 2019:651067.
• 29. Pande A, Patiyal S, Lathwal A. et al.  Computing wide range of protein/peptide features from their sequence and structure. BioRxiv  2019:599126. [Google Scholar]
• 30. Chen Z, Zhao P, Li F. et al.  Ifeature: a python package and web server for features extraction and selection from protein and peptide sequences. Bioinformatics  2018;34:2499–502. 10.1093/bioinformatics/bty140. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 31. Vens C, Rosso M-N, Danchin EGJ. Identifying discriminative classification-based motifs in biological sequences. Bioinformatics  2011;27:1231–8. 10.1093/bioinformatics/btr110. [DOI] [PubMed] [Google Scholar]
• 32. Kumar M, Raghava GPS. et al.  Identification of dna-binding proteins using support vector machines and evolutionary profiles. J Eur Psychol Stud  2007;8:463–0. 10.1186/1471-2105-8-463. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 33. Osorio D, Rondón-Villarreal P, Torres R. Peptides: a package for data mining of antimicrobial peptides. Small  2015;12:44–444. [Google Scholar]
• 34. Mall R, Suykens JAK. Sparse reductions for fixed-size least squares support vector machines on large scale data. In: Pei J, Tseng VS, Cao L, Motoda H, Xu G (eds.), Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, Goldcoast, Australia, 2013, pages 161–73. 10.1007/978-3-642-37453-1_14. [DOI] [Google Scholar]
• 35. Cortes C, Vapnik V. Support-vector networks. Mach Learn  1995;20:273–97. 10.1007/BF00994018. [DOI] [Google Scholar]
• 36. Friedman JH. Greedy function approximation: a gradient boosting machine. Ann Stat  2001;29:1189–232. 10.1214/aos/1013203451. [DOI] [Google Scholar]
• 37. Mall R, Cerulo L, Garofano L. et al.  Rgbm: regularized gradient boosting machines for identification of the transcriptional regulators of discrete glioma subtypes. Nucleic Acids Res  2018;46:e39–9. 10.1093/nar/gky015. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 38. Rosenblatt F. The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev  1958;65:386–408. 10.1037/h0042519. [DOI] [PubMed] [Google Scholar]
• 39. Breiman L. Random forests. Mach Learn  2001;45:5–32. 10.1023/A:1010933404324. [DOI] [Google Scholar]
• 40. The UniProt Consortium . UniProt: the universal protein knowledgebase in 2023. Nucleic Acids Res  2022;51:D523–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 41. Jeffrey Cole T, Brewer MS. Toxify: a deep learning approach to classify animal venom proteins. PeerJ  2019;7:e7200. 10.7717/peerj.7200. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 42. Pan X, Zuallaert J, Wang X. et al.  Toxdl: deep learning using primary structure and domain embeddings for assessing protein toxicity. Bioinformatics  2020;36:5159–68. [DOI] [PubMed] [Google Scholar]
• 43. Wei L, Ye X, Xue Y. et al.  Atse: a peptide toxicity predictor by exploiting structural and evolutionary information based on graph neural network and attention mechanism. Brief Bioinform  2021;22:bbab041. [DOI] [PubMed] [Google Scholar]
• 44. Wei L, Ye X, Sakurai T. et al.  Toxibtl: prediction of peptide toxicity based on information bottleneck and transfer learning. Bioinformatics  2022;38:1514–24. 10.1093/bioinformatics/btac006. [DOI] [PubMed] [Google Scholar]
• 45. Morozov V, Rodrigues CHM, Ascher DB. Csm-toxin: a web-server for predicting protein toxicity. Pharmaceutics  2023;15:431. 10.3390/pharmaceutics15020431. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 46. Velickovic P, Cucurull G. et al.  Graph Attention Networks.  6th International Conference on Learning Representations (ICLR), 2018;30:1–12. [Google Scholar]
• 47. Mall R, Elbasir A, Almeer H. et al.  A modeling framework for embedding-based predictions for compound–viral protein activity. Bioinformatics  2021;37:2544–55. 10.1093/bioinformatics/btab130. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 48. Vaswani A, Shazeer N, Parmar N. et al.  Attention is all you need. Adv Neural Inf Process Syst  2017;30:1–11. [Google Scholar]
• 49. Brandes N, Ofer D, Peleg Y. et al.  Proteinbert: a universal deep-learning model of protein sequence and function. Bioinformatics  2022;38:2102–10. 10.1093/bioinformatics/btac020. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 50. Devlin J, Chang M-W, Lee K. et al.  Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of naacL-HLT  2019;1:2. [Google Scholar]
• 51. Lin Z, Akin H, Rao R. et al.  Evolutionary-scale prediction of atomic-level protein structure with a language model. Science  2023;379:1123–30. 10.1126/science.ade2574. [DOI] [PubMed] [Google Scholar]
• 52. Li F-Z, Amini AP, Yue Y. et al.  Feature reuse and scaling: Understanding transfer learning with protein language models  bioRxiv. 2024:2024–02.
• 53. Ke G, Meng Q, Finley T. et al.  Lightgbm: a highly efficient gradient boosting decision tree. Adv Neural Inf Process Syst  2017;30:1–9. [Google Scholar]
• 54. Chen Tianqi and GuestrinCarlos. Xgboost: A scalable tree boosting system. In: Smola A, Aggarwal C, Shen D, Rastogi R (eds.), Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, ACM Digital Library, San Francisco California, USA, 2016, pp 785–794.
• 55. Matthews BW. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochim Biophys Protein Struct  1975;405:442–51. 10.1016/0005-2795(75)90109-9. [DOI] [PubMed] [Google Scholar]
• 56. Rawi R, Mall R, Kunji K. et al.  Parsnip: sequence-based protein solubility prediction using gradient boosting machine. Bioinformatics  2018;34:1092–8. 10.1093/bioinformatics/btx662. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 57. Khurana S, Rawi R, Kunji K. et al.  Deepsol: a deep learning framework for sequence-based protein solubility prediction. Bioinformatics  2018;34:2605–13. 10.1093/bioinformatics/bty166. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 58. Uniprot: the universal protein knowledgebase in 2023. Nucleic Acids Res  2023;51:D523–31. 10.1093/nar/gkac1052. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 59. Limin F, Niu B, Zhu Z. et al.  Cd-hit: accelerated for clustering the next-generation sequencing data. Bioinformatics  2012;28:3150–2. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 60. Addou S, Rentzsch R, Lee D. et al.  Domain-based and family-specific sequence identity thresholds increase the levels of reliable protein function transfer. J Mol Biol  2009;387:416–30. 10.1016/j.jmb.2008.12.045. [DOI] [PubMed] [Google Scholar]
• 61. Kleandrova VV, Ruso JM, Speck-Planche A. et al.  Enabling the discovery and virtual screening of potent and safe antimicrobial peptides. Simultaneous prediction of antibacterial activity and cytotoxicity. ACS Comb Sci  2016;18:490–8. 10.1021/acscombsci.6b00063. [DOI] [PubMed] [Google Scholar]
• 62. Gogoladze G, Grigolava M, Vishnepolsky B. et al.  Dbaasp: database of antimicrobial activity and structure of peptides. FEMS Microbiol Lett  2014;357:63–8. 10.1111/1574-6968.12489. [DOI] [PubMed] [Google Scholar]
• 63. Devlin J, Chang M-W, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding.  NAACL-HLT  2019;1:4171–86. [Google Scholar]
• 64. Wolf T, Debut L, Sanh V. et al.  Huggingface’s transformers: State-of-the-art natural language processing  arXiv preprint arXiv:1910.03771. 2019.
• 65. Kinga D, Adam JB. A method for stochastic optimization. In: Bengio Y, Lecunn Y (eds.), International conference on learning representations (ICLR), Vol. 5. San Diego, California, OpenReview, 2015, 6. [Google Scholar]
• 66. Elbasir A, Moovarkumudalvan B, Kunji K. et al.  Deepcrystal: a deep learning framework for sequence-based protein crystallization prediction. Bioinformatics  2019;35:2216–25. 10.1093/bioinformatics/bty953. [DOI] [PubMed] [Google Scholar]
• 67. Palotti J, Mall R, Aupetit M. et al.  Benchmark on a large cohort for sleep-wake classification with machine learning techniques. NPJ Digital Med  2019;2:50. 10.1038/s41746-019-0126-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 68. Rawi R, Mall R, Chen-Hsiang Shen S. et al.  Accurate prediction for antibody resistance of clinical hiv-1 isolates. Sci Rep  2019;9:1–14696. 10.1038/s41598-019-50635-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 69. Mall R, Langone R, Suykens JAK. Multilevel hierarchical kernel spectral clustering for real-life large scale complex networks. PloS One  2014;9:e99966. 10.1371/journal.pone.0099966. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 70. Opitz D, Maclin R. Popular ensemble methods: an empirical study. J Artif Intell Res  1999;11:169–98. 10.1613/jair.614. [DOI] [Google Scholar]
• 71. Gysi DM, Do Valle Í, Zitnik M. et al.  Network medicine framework for identifying drug-repurposing opportunities for covid-19. Proc Natl Acad Sci  2021;118:e2025581118. 10.1073/pnas.2025581118. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 72. Jumper J, Evans R, Pritzel A. et al.  Highly accurate protein structure prediction with alphafold. Nature  2021;596:583–9. 10.1038/s41586-021-03819-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 73. Kiefer F, Arnold K, Künzli M. et al.  The swiss-model repository and associated resources. Nucleic Acids Res  2009;37:D387–92. 10.1093/nar/gkn750. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 74. Pieper U, Webb BM, Barkan DT. et al.  Modbase, a database of annotated comparative protein structure models, and associated resources. Nucleic Acids Res  2010;39:D465–74. 10.1093/nar/gkq1091. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 75. Laskowski R, MacArthur MW, Thornton JM.  International Tables for Crystallography, Chapter 21.4, 2012;F:684–7. 10.1107/97809553602060000882. [DOI] [Google Scholar]
• 76. Zdrazil B, Felix E, Hunter F. et al.  The chembl database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods. Nucleic Acids Res  2024;52:D1180–92. 10.1093/nar/gkad1004. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 77. Patel CN, Mall R, Bensmail H. Ai-driven drug repurposing and binding pose meta dynamics identifies novel targets for monkeypox virus. J Infect Public Health  2023;16:799–807. 10.1016/j.jiph.2023.03.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 78. Jorgensen WL, Tirado-Rives J. The opls force field for proteins. Energy minimizations for crystals of cyclic peptides and crambin. J Am Chem Soc  1988;110:1657–66. 10.1021/ja00214a001. [DOI] [PubMed] [Google Scholar]
• 79. Jorgensen WL, Maxwell DS, Tirado-Rives J. Development and testing of the opls all-atom force field on conformational energetics and properties of organic liquids. J Am Chem Soc  1996;118:11225–36. 10.1021/ja9621760. [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

VISH_Pred_Supplementary_bbae270

Data Availability Statement

All the data required to train the model are available via https://bitbucket.org/ascherslab/csm-toxin/src/master/data_processing/ (accessed on 17 January 2023) and VISH-Pred model is available as a web-server for testing at: http://ec2-35-170-123-194.compute-1.amazonaws.com:7860/.