SpanSeq: Similarity-based sequence data splitting method for improved development and assessment of deep learning projects (2024)

1 Introduction

The value of deep learning models resides in their ability to predict accurately on previously unseen data. To achieve this generalization capacity, they rely on their considerable plasticity, which allows them to learn underlying patterns in the data available, regardless of their complexity. Moreover, this potential allows them to memorize individual samples, perfectly fitting input with output of the training data, while decreasing the loss of the training set. The capacity to memorize by neural networks should not be underestimated, as for this they have shown to able to fit random data perfectly Zhang etal. (2021). On real datasets, it has been observed that this issue can be caused by noisy data Arpit etal. (2017) and out-of-distribution samples Carlini etal. (2019).Other factors include deeper architectures in neural networks Tirumala etal. (2022); Zhang etal. (2019); Arpit etal. (2017); Carlini etal. (2019), and repeated exposure to the same samples Carlini etal. (2019), which can come from the epoch iteration during the training process, or from duplicated samples Carlini etal. (2022).
Although some of these memorization factors are shared with overfitting, these two phenomena are different, and carry different consequences. While the first defines particular samples (mapping input to output), the latter describes the fitting of the particularities of the training set, considering them as underlying patterns of the true distribution of data. This will be reflected at the performance difference of the model on training and test sets, which will be notably lower on the latter Tetko etal. (1995).When that happens, the value of the model is dubious, as its generalization capacity is not optimal. Other differences have been found between memorization and overfitting, as the first happening earlier during the training process Tirumala etal. (2022), and methods like regularization, used to control overfitting, showing very limited effect on it Carlini etal. (2019).Nevertheless, the effect of memorization on the efficiency of a deep learning model is ambiguous Chatterjee (2018). Being able to memorize out-of-distribution samples can be beneficial, particularly with certain data distributions Feldman and Zhang (2020); Feldman (2020). But it also provoke complications, as with privacy issues in language models Carlini etal. (2022); Lee etal. (2021), or by confusing generalization with memorization, overestimating the efficiency of a model Elangovan etal. (2021).

A long-established procedure when developing a deep learning model is to split the available data in different sets, which are employed for different purposes. Frequently, for fitting the model (train set), tuning its hyperparameters (training and development or validation set) and evaluating its performance (test set) Hastie etal. (2009). Thus, train and validation sets are used during the development of the model, using the latter to evaluate the model in pursuance of choosing a set of hyperparameters that provides a better model. Meanwhile, the test set will be used to evaluate the performance of the model Westerhuis etal. (2008), to have an estimation of how its performance will be once deployed. Multiple variants of this rigid split scheme have been proposed, handling issues as related to single splits, unequal distribution of targets and representation of the true distribution of the data Kohavi (1995); Daszykowski etal. (2002); Harrington (2018); Cawley and Talbot (2010); Xu and Goodacre (2018). However, using a hold out set for testing the performance of the model has been kept as the standard evaluation of its efficiency Westerhuis etal. (2008).The split between data for training/validation and testing should ensure a truthful evaluation of the generalization capacity of the model, without being affected by its memorization capacity Hastie etal. (2009); Westerhuis etal. (2008).The most common strategy for distributing the available data in different splits is to do it randomly (with respect to the input of the model), in an attempt to avoid any influence from the developer’s previous knowledge of the dataset. This strategy is based on the assumption the data is independent and identically distributed (i.i.d.) Vapnik (1995), which is not always the case, specially with certain types of data. Duplicates and similarity among samples populate the existing databases, and they spread among the train/validation/test sets if created by random splitting. When that happens, similar points appear in both training and test sets, inducing data leakage. This event has been found when developing and testing deep learning models on images Tampu etal. (2022), text Elangovan etal. (2021); Søgaard etal. (2020) and code Allamanis (2019), meaning that the prediction on the test can be solved by memorization, leading to an overestimation of their capacity to generalize. However, data leakage through similarity is not restricted to those types of data, as the features that make them prone to data leakage can be found elsewhere, including in biological sequential data, like genes, proteins or genomes, due to the evolutionary relationships that exist among them, and the redundancy in their databases Hobohm etal. (1992).

In fact, it is well known that closeness between biological sequences tends to lead to similarity of their phenotypes Lund etal. (1997). This has been the principle used on alignment-based prediction methods Pearson (2013), but also has been object of concern with as it brings data leakage when creating machine learning models. Some strategies have been used in the last years to address the split of datasets of biological sequences, mainly built around the concepts of data reduction and data partitioning. The first clusters the data around representative sample points of the data itself, which are separate above a certain threshold of similarity. It has several benefits, such as solving the issue of similarity inside sets Allamanis (2019). However, it also brings consequences, as choosing a representative data point over the rest can bias the dataset, or the amount of data left might be insufficient for proper learning, as well as the possible beneficial effects similar samples can bring to training (as adversarial examples Goodfellow etal. (2014)). The second, data partitioning, groups similar sequences in clusters, which will be distributed among the partitions, avoiding similar samples to be in different partitions. Those two strategies are not exclusive, as the first can be used for removing duplicates, and the second for removing any data leakage from less similar sequences. For the research in this paper, we have chosen to evaluate on the latter, as it is a feasible step no matter the quantity of data available.

Grouping sequences by similarity is a step common in both reduction and partition strategies. For measuring similarity in biological sequences, traits from pairwise alignment methods Needleman and Wunsch (1970); Gotoh (1982) have been largely used, as it is an established method on the field. Particularly, percent identity due to the ease of its interpretation. However, these methods have a complexity of $\mathcal{O}(l_{1}l_{2})$, where $l_{1},l_{2}$ are the lengths of the two sequences aligned. With this complexity, calculating all the distances with pairwise alignment of a large dataset (as required for deep learning) can easily become infeasible, especially as the length of the sequences grows. In order to address large datasets, most of the clustering strategies for biological sequences attempt to reduce the amount of alignments, like CD-HIT Fu etal. (2012), uCLUST Prasad etal. (2015), HHblits Remmert etal. (2012), and MMseqs Hauser etal. (2016), as per data reduction. While some of those methods might be adequate for data reduction, due to using the representatives of each cluster for the similarity threshold, they are not suitable for data partition. A few methods have recently addressed this issue, such as attempting to create appropriate train and test sets of biological sequences Petti and Eddy (2022), split molecular databases to reduce data leakage (DataSAIL Joeres etal. (2023)), or partition databases using graph-based approaches (GraphPart Teufel etal. (2023)). Although all of them split the data necessary for deep learning development at a reasonable time, as alignment-based approaches its usefulness is limited by sequence lengths.

In this work we present SpanSeq, an alignment-free approach to data partitioning for deep learning applications. With it, we are able to cluster and partition datasets in the order of $10^{5}$ biological sequences, as genes, proteins or even genomes, while using an all-vs-all clustering calculation. Moreover, we apply this new method to the state-of-the-art protein subcellular location model DeepLoc AlmagroArmenteros etal. (2017); Thumuluri etal. (2022), proving the importance of similarity-based data splitting not only during model assessment, but through the whole model development process.

2 Materials and Methods

2.3 Similarity-based partition effect on DeepLoc

To evaluate the effect of similarity-based partitioning, the development of the deep learning model DeepLoc 1.0 AlmagroArmenteros etal. (2017) was reproduced (hyperparameter selection, training, model assessment). This deep neural network predicts the subcellular location of eukaryotic proteins, using a combination of convolutional, LSTM, and attention layers. Although at the time of the SpanSeq research DeepLoc 2.0 Thumuluri etal. (2022) had been released, it was decided to evaluate on the first version, as the newest uses a pre-trained model, so that part of the model could not be evaluated with different similarity splits.

As DeepLoc works on amino acid sequences, a fixed threshold of 0.3 similarity was chosen, below which two sequences would not be considered close enough to be prone to data leakage. This number was chosen based on previous studies of the relationship between the amino acid sequence and general structure features Sander and Schneider (1991); Lund etal. (1997), and its usage in protein-based deep learning models AlmagroArmenteros etal. (2017). However, the strategy for selecting a threshold to define data leakage by similarity is outside the scope of this research.

2.3.1 Datasets

The dataset of DeepLoc 2.0 Thumuluri etal. (2022) was used for developing DeepLoc 1.0 AlmagroArmenteros etal. (2017), as it was newer and larger Thumuluri etal. (2022). Proteins with multiple locations, as well as the ones longer than 1,000 amino acids, were dropped due to DeepLoc 1.0’s inability to classify these, unlike DeepLoc 2.0, leaving a dataset of 19,171 sequences. In order to analyze the effect of data similarity partition during the development of a deep learning model, the dataset created was split in different ways:

• •

  Hold Out Test Set: The dataset was originally partitioned in six subsets using SpanSeq with Mash, so there was a maximum 0.3 similarity between sequences in different sets. From that split, 1 set was selected randomly to be the hold out test set, while the other five were merged to be the development set.

• •

  Development Set: This set was divided in five splits: one for testing and four for training and validation set. However, four different methods for partitioning were used, each of which with different approaches towards similarity between splits.

  • –

    SpanSeq (Mash) Split: The development set was split using SpanSeq, with Mash Ondov etal. (2016) for calculating the distance among sequences. Thus, there is no pair of sequences more than 0.3 similar (mash distance) in two different partitions.

  • –

    SpanSeq (Pairwise Alignment) Split: The development set is split using SpanSeq, with global pairwise alignment (with ggsearch36 Pearson (2016)) for calculating the distance among sequences. Thus, there are no pair of sequences with more than 30% identity in two different partitions.

  • –

    Random Split: The development set is split randomly, so there is no control of where similar sequences go.

  • –

    Increased Similarity Split: The development set is split with similar sequences being spread in different partitions, so that the similarity is increased between them. This was done by inverting the distance measure between sequences, so that closely related sequences were treated as distantly related sequences and vice.verse, thus spreading them into different partitions.

2.3.2 Development of DeepLoc with different datasets

The generalization of MCC (Matthews Correlation Coefficient) Matthews (1975) for $K$ categories known as Gorodkin measure Gorodkin (2004) was used as performance measure.

With each of the four different dataset splits, SpanSeq-Mash, SpanSeq-Alignment, Random, Increased Similarity, the same process was followed for developing DeepLoc. The hyperparameter selection was performed using the four training/validation sets with Bayesian hyperparameter optimization using SigOpt Hayes etal. (2019), with the four different datasets having the same parameters for hyperparameter selection (amount of combination tried, hyperparameter’s ranges explored).

See Also

Identification and Molecular Characterization of a Novel Carlavirus Infecting Chrysanthemum morifolium in China.

Each of the four different dataset splits and combination of hyperparameters was used for training a DeepLoc model. In order to avoid overfitting, each model was trained four times, where each time three partitions were used for training while one was used as a validation set. The training lasted for 800 epochs, and the model iteration with best performance on the validation set was selected, in order to assess the effect of similarity on classic methods like early-stopping.

The hyperparameter optimization and training were performed on one Node of 2 Tesla V100 16 GB GPUs.

3 Results

3.1 SpanSeq features

3.1.1 Distance measure

The correlation between the distance measures based on k-mers used by SpanSeq and the identity from global alignment can be visualized on the protein (Figure 1(a)) and gene (Figures 1(b)-1(f)) datasets. While mash, cosine, k-mer inverse coverage and Szymkiewicz-Simpson distances seem to follow identity, jaccard distance gave values higher than the identity predicted on the same pair of sequences. Further exploration of the relation between the difference between k-mer distances and sequence lengths was performed (Supplementary Data, Figure S3, as well as the effects of k-mer and minimizer size, although an in-depth study of them is outside the scope of this research.

3.1.2 Taxonomic relations in clustering

The clustering results of SpanSeq on the RefSeq database O’Leary etal. (2016) ($10^{4}$ complete genomes) show differences between the distance measures and the taxonomic relations among the genomes clustered (Figure 2). The cosine, Szymkiewicz–Simpson and k-mer inverse coverage distances show similar distributions for the five NCBI taxonomic ranks (Species, Genus, Family, Order and Class), as for the last four the distance is mainly almost 1, while in the species rank the majority of the distances are between 0.2 and 0.4. With the Jaccard distance, we observe that the distribution of distances for species is higher, which is a similar behavior as seen in Figure 1(f). Due to the distribution of distances at the other taxonomic levels, it is difficult to compare the behavior of Jaccard on those with the other distances from KMA Clausen etal. (2018). With Mash distance, the distributions on the taxonomic ranks differ considerably from the other distances. Most of the distances at the species rank are below 0.05, while at Genus, family and order ranks most of the distances are between 0.4 and 1, although there are plenty of distances close to 0.

3.1.3 Performance evaluation

The evaluation of the performance of SpanSeq was done on the subsets of the genomic dataset created from RefSeq, in order to evaluate the method not only in large datasets (from $10^{1}$ to $10^{5}$), but also on sequences with lengths large enough to be prohibitive for most alignment methods. The results show that SpanSeq was able to partition datasets in a sensible time relative to the size of the datasets (Supplementary Data, Figure S4) without using the Hobohm1 strategy, while using different strategies depending on the size of the dataset (minimizers/prefixes).

3.2 Similarity partitioning

3.2.1 Hyperparameter selection

During the hyperparameter selection, the performance during the cross-validation process in the different dataset splits is directly correlated with the similarity between the folds (Supplementary Data, Table S1). The hyperparameter configuration for each of the dataset splits is also different, although SigOpt Hayes etal. (2019) was provided with the same range of hyperparameters for each split of data (Supplementary Data, Table S1).

3.2.2 Training process

The training curves of the DeepLoc algorithm show clear differences among partition strategies. In the increased similarity and random split training sets, it is reached a Gorodkin value higher than 0.9 before 100 (Figure 3(a)) and 200 epochs (Figure 3(b)), respectively, followed by a slight plateau of its improvement. The Gorodkin measures of the validation set follows a similar behavior, but 0.2 points below. In the case of the two splits made with SpanSeq, the training curves draw very different dynamics, as they increase sharply during the first epochs, but after which they keep incrementing slowly from there without reaching 0.9 during the 800 epochs trained (Figures 3(c), 3(d)). The Gorodkin measure curve in the validation set also differs from the other two splits of the dataset, as the Gorodkin values in both SpanSeq splits seem to decrease after the 100th epoch, unlike the training’s done without SpanSeq, which plateau or even increase (Figures 3(a), 3(b)).We also saw differences on the location of the epochs where the performance of the validation set is higher among the different ways of splitting the dataset. Those epochs appear certainly much earlier during the training of SpanSeq split datasets (vertical lines in 3; Supplementary Material, Figure S5).

3.2.3 Model evaluation

When evaluating the performance of DeepLoc (Figure 4; Supplementary Material, Table S2), we observe the same differences seen in Figure 3 between splitting methods. However, the performance on the test and test holdout show differences between them depending on the split method. Using SpanSeq, the performances on the test and test holdout are almost equal, and so on the validation set. But when splitting randomly or increasing the similarity among sets, the validation and test set have a notably higher Gorodkin measures than on the test holdout, which are slightly smaller than on the models developed with SpanSeq.

4 Discussion

5 Conclusion

In this work, we present SpanSeq, a dataset partition method specially created for the development of deep learning models. Unlike other clustering methods for biological sequences Fu etal. (2012), SpanSeq uses a clustering method based on k-mer distances and all-vs-all clustering, to assess data leakage from similarity during data partition. While it provides balanced partitions with respect to partition size as well as the option to minimize class imbalance. The method is able to handle large datasets of biological sequences, no matter its length (proteins, genes and genomes).

Furthermore, we prove the need to use SpanSeq over random splits to limit the similarity between partitions. First, because it prevents to report overconfident performance of the test set due to being affected by the memorization capacity of the model. Second, to protect methods such as hyperparameter selection or early-stopping, based on the performance on validation sets, so they benefit models with generalization capacity. And last, as it reduces largely the resources necessary for developing a deep learning model, without compromising the generalization capacity of the model.

6 Data Availability

7 Supplementary Data

Supplementary data is available at the file Supplementary Material

8 Funding

This study was supported by the European Union’s Horizon 2020 research and innovation program under VEO grant agreement No. 874735.

9 Conflict of interests

Jose Juan Almagro Armenteros is an employee of Bristol Myers Squibb Company at the time of the publication; however, that did not influence the research in any way.

References

• Zhang etal. [2021]Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.Understanding deep learning (still) requires rethinking generalization.Communications of the ACM, 64(3):107–115, 2021.
• Arpit etal. [2017]Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, MaxinderS Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, etal.A closer look at memorization in deep networks.In International conference on machine learning, pages 233–242. PMLR, 2017.
• Carlini etal. [2019]Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song.The secret sharer: Evaluating and testing unintended memorization in neural networks.In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284, 2019.
• Tirumala etal. [2022]Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan.Memorization without overfitting: Analyzing the training dynamics of large language models.Advances in Neural Information Processing Systems, 35:38274–38290, 2022.
• Zhang etal. [2019]Chiyuan Zhang, Samy Bengio, Moritz Hardt, MichaelC Mozer, and Yoram Singer.Identity crisis: Memorization and generalization under extreme overparameterization.arXiv preprint arXiv:1902.04698, 2019.
• Carlini etal. [2022]Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang.Quantifying memorization across neural language models.arXiv preprint arXiv:2202.07646, 2022.
• Tetko etal. [1995]IgorV Tetko, DavidJ Livingstone, and AlexanderI Luik.Neural network studies. 1. comparison of overfitting and overtraining.Journal of chemical information and computer sciences, 35(5):826–833, 1995.
• Chatterjee [2018]Satrajit Chatterjee.Learning and memorization.In International conference on machine learning, pages 755–763. PMLR, 2018.
• Feldman and Zhang [2020]Vitaly Feldman and Chiyuan Zhang.What neural networks memorize and why: Discovering the long tail via influence estimation.Advances in Neural Information Processing Systems, 33:2881–2891, 2020.
• Feldman [2020]Vitaly Feldman.Does learning require memorization? a short tale about a long tail.In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954–959, 2020.
• Lee etal. [2021]Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.Deduplicating training data makes language models better.arXiv preprint arXiv:2107.06499, 2021.
• Elangovan etal. [2021]Aparna Elangovan, Jiayuan He, and Karin Verspoor.Memorization vs. generalization: quantifying data leakage in NLP performance evaluation.arXiv preprint arXiv:2102.01818, 2021.
• Hastie etal. [2009]Trevor Hastie, JeromeH Friedman, and Robert Tibshirani.The elements of statistical learning: data mining, inference, and prediction, volume2.Springer, 2009.
• Westerhuis etal. [2008]JohanA Westerhuis, HuubCJ Hoefsloot, Suzanne Smit, DanielJ Vis, AgeK Smilde, EwoudJJ van Velzen, JohnPM van Duijnhoven, and FerdiA van Dorsten.Assessment of PLSDA cross validation.Metabolomics, 4:81–89, 2008.
• Kohavi [1995]Ron Kohavi.A study of cross-validation and bootstrap for accuracy estimation and model selection.In The International Joint Conference on AI, volume14, pages 1137–1145. Montreal, Canada, 1995.
• Daszykowski etal. [2002]Michal Daszykowski, Beata Walczak, and DLMassart.Representative subset selection.Analytica chimica acta, 468(1):91–103, 2002.
• Harrington [2018]Peter deBoves Harrington.Multiple versus single set validation of multivariate models to avoid mistakes.Critical reviews in analytical chemistry, 48(1):33–46, 2018.
• Cawley and Talbot [2010]GavinC Cawley and NicolaLC Talbot.On over-fitting in model selection and subsequent selection bias in performance evaluation.The Journal of Machine Learning Research, 11:2079–2107, 2010.
• Xu and Goodacre [2018]Yun Xu and Royston Goodacre.On splitting training and validation set: a comparative study of cross-validation, bootstrap and systematic sampling for estimating the generalization performance of supervised learning.Journal of analysis and testing, 2(3):249–262, 2018.
• Vapnik [1995]VladimirN Vapnik.The nature of statistical learning theory.Springer, 1995.
• Tampu etal. [2022]IulianEmil Tampu, Anders Eklund, and Neda Haj-Hosseini.Inflation of test accuracy due to data leakage in deep learning-based classification of oct images.Scientific Data, 9(1):580, 2022.
• Søgaard etal. [2020]Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova.We need to talk about random splits.arXiv preprint arXiv:2005.00636, 2020.
• Allamanis [2019]Miltiadis Allamanis.The adverse effects of code duplication in machine learning models of code.In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pages 143–153, 2019.
• Hobohm etal. [1992]Uwe Hobohm, Michael Scharf, Reinhard Schneider, and Chris Sander.Selection of representative protein data sets.Protein Science, 1(3):409–417, 1992.
• Lund etal. [1997]Ole Lund, Kenneth Frimand, Jan Gorodkin, Henrik Bohr, Jakob Bohr, Jan Hansen, and Søren Brunak.Protein distance constraints predicted by neural networks and probability density functions.Protein Engineering, 10(11):1241–1248, 1997.
• Pearson [2013]WilliamR Pearson.An introduction to sequence similarity (“hom*ology”) searching.Current protocols in bioinformatics, 42(1):3–1, 2013.
• Goodfellow etal. [2014]IanJ Goodfellow, Jonathon Shlens, and Christian Szegedy.Explaining and harnessing adversarial examples.arXiv preprint arXiv:1412.6572, 2014.
• Needleman and Wunsch [1970]SaulB Needleman and ChristianD Wunsch.A general method applicable to the search for similarities in the amino acid sequence of two proteins.Journal of molecular biology, 48(3):443–453, 1970.
• Gotoh [1982]Osamu Gotoh.An improved algorithm for matching biological sequences.Journal of molecular biology, 162(3):705–708, 1982.
• Fu etal. [2012]Limin Fu, Beifang Niu, Zhengwei Zhu, Sitao Wu, and Weizhong Li.CD-HIT: accelerated for clustering the next-generation sequencing data.Bioinformatics, 28(23):3150–3152, 2012.
• Prasad etal. [2015]DVenkatavara Prasad, Sathya Madhusudanan, and Suresh Jaganathan.uCLUST – a new algorithm for clustering unstructured data.ARPN Journal of Engineering and Applied Sciences, 10(5):2108–2117, 2015.
• Remmert etal. [2012]Michael Remmert, Andreas Biegert, Andreas Hauser, and Johannes Söding.HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment.Nature methods, 9(2):173–175, 2012.
• Hauser etal. [2016]Maria Hauser, Martin Steinegger, and Johannes Söding.MMseqs software suite for fast and deep clustering and searching of large protein sequence sets.Bioinformatics, 32(9):1323–1330, 2016.
• Petti and Eddy [2022]Samantha Petti and SeanR Eddy.Constructing benchmark test sets for biological sequence analysis using independent set algorithms.PLOS Computational Biology, 18(3):e1009492, 2022.
• Joeres etal. [2023]Roman Joeres, DavidB Blumenthal, and OlgaV Kalinina.DataSAIL: Data splitting against information leakage.bioRxiv, page 2023.11.15.566305, 2023.
• Teufel etal. [2023]Felix Teufel, MagnúsHalldór Gíslason, JoséJuan AlmagroArmenteros, AlexanderRosenberg Johansen, Ole Winther, and Henrik Nielsen.GraphPart: hom*ology partitioning for biological sequence analysis.NAR Genomics and Bioinformatics, 5(4):lqad088, 2023.
• AlmagroArmenteros etal. [2017]JoséJuan AlmagroArmenteros, CasperKaae Sønderby, SørenKaae Sønderby, Henrik Nielsen, and Ole Winther.DeepLoc: prediction of protein subcellular localization using deep learning.Bioinformatics, 33(21):3387–3395, 2017.
• Thumuluri etal. [2022]Vineet Thumuluri, JoséJuan AlmagroArmenteros, AlexanderRosenberg Johansen, Henrik Nielsen, and Ole Winther.DeepLoc 2.0: multi-label subcellular localization prediction using protein language models.Nucleic acids research, 50(W1):W228–W234, 2022.
• Ondov etal. [2016]BrianD Ondov, ToddJ Treangen, Páll Melsted, AdamB Mallonee, NicholasH Bergman, Sergey Koren, and AdamM Phillippy.Mash: fast genome and metagenome distance estimation using minhash.Genome biology, 17(1):1–14, 2016.
• Clausen etal. [2018]PhilipTLC Clausen, FrankM Aarestrup, and Ole Lund.Rapid and precise alignment of raw reads against redundant databases with KMA.BMC bioinformatics, 19:1–8, 2018.
• Mölder etal. [2021]Felix Mölder, KimPhilipp Jablonski, Brice Letcher, MichaelB Hall, ChristopherH Tomkins-Tinch, Vanessa Sochat, Jan Forster, Soohyun Lee, SvenO Twardziok, Alexander Kanitz, etal.Sustainable data analysis with Snakemake.F1000Research, 10:33, 2021.
• Clausen etal. [2016]PhilipTLC Clausen, EaZankari, FrankM Aarestrup, and Ole Lund.Benchmarking of methods for identification of antimicrobial resistance genes in bacterial whole genome data.Journal of Antimicrobial Chemotherapy, 71(9):2484–2488, 2016.
• Li [2016]Heng Li.Minimap and miniasm: fast mapping and de novo assembly for noisy long sequences.Bioinformatics, 32(14):2103–2110, 2016.
• Ester etal. [1996]Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, etal.A density-based algorithm for discovering clusters in large spatial databases with noise.In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, volume96, pages 226–231. AAAI Press, 1996.
• Clausen [2023]PhilipTLC Clausen.Scaling neighbor joining to one million taxa with dynamic and heuristic neighbor joining.Bioinformatics, 39(1):btac774, 2023.
• Gan and Tao [2015]Junhao Gan and Yufei Tao.DBSCAN revisited: Mis-claim, un-fixability, and approximation.In Proceedings of the 2015 ACM SIGMOD international conference on management of data, pages 519–530, 2015.
• Hübscher and Glover [1994]Roland Hübscher and Fred Glover.Applying tabu search with influential diversification to multiprocessor scheduling.Computers & operations research, 21(8):877–884, 1994.
• Bortolaia etal. [2020]Valeria Bortolaia, RolfS Kaas, Etienne Ruppe, MarilynC Roberts, Stefan Schwarz, Vincent Cattoir, Alain Philippon, RosaL Allesoe, AnaRita Rebelo, AlfredFerrer Florensa, etal.Resfinder 4.0 for predictions of phenotypes from genotypes.Journal of Antimicrobial Chemotherapy, 75(12):3491–3500, 2020.
• O’Leary etal. [2016]NualaA O’Leary, MathewW Wright, JRodney Brister, Stacy Ciufo, Diana Haddad, Rich McVeigh, Bhanu Rajput, Barbara Robbertse, Brian Smith-White, Danso Ako-Adjei, etal.Reference sequence (RefSeq) database at NCBI: current status, taxonomic expansion, and functional annotation.Nucleic acids research, 44(D1):D733–D745, 2016.
• Pearson [2016]WilliamR Pearson.Finding protein and nucleotide similarities with fasta.Current protocols in bioinformatics, 53(1):3–9, 2016.
• Sander and Schneider [1991]Chris Sander and Reinhard Schneider.Database of hom*ology-derived protein structures and the structural meaning of sequence alignment.Proteins: Structure, Function, and Bioinformatics, 9(1):56–68, 1991.
• Rost [1999]Burkhard Rost.Twilight zone of protein sequence alignments.Protein engineering, 12(2):85–94, 1999.
• Matthews [1975]BrianW Matthews.Comparison of the predicted and observed secondary structure of t4 phage lysozyme.Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451, 1975.
• Gorodkin [2004]Jan Gorodkin.Comparing two k-category assignments by a k-category correlation coefficient.Computational biology and chemistry, 28(5-6):367–374, 2004.
• Hayes etal. [2019]Patrick Hayes, Dan Anderson, Bolong Cheng, TaylorJackle Spriggs, Alexandra Johnson, and Michael McCourt.SigOpt documentation.Technical Report SO-12/14 – Revision 1.07, SigOpt, Inc., 2019.URL https://sigopt.com/docs.
• Fay and Proschan [2010]MichaelP Fay and MichaelA Proschan.Wilcoxon-mann-whitney or t-test? on assumptions for hypothesis tests and multiple interpretations of decision rules.Statistics surveys, 4:1, 2010.
• Moreno etal. [2002]Edgardo Moreno, Axel Cloeckaert, and Ignacio Moriyón.Brucella evolution and taxonomy.Veterinary microbiology, 90(1-4):209–227, 2002.
• Göttke etal. [2000]MarkusU Göttke, CarloA Fallone, AlanN Barkun, Konstanze Vogt, Vivian Loo, Matthias Trautmann, JianZ Tong, ThanhN Nguyen, Toby Fainsilber, HelmutH Hahn, etal.Genetic variability determinants of helicobacter pylori: influence of clinical background and geographic origin of isolates.The Journal of infectious diseases, 181(5):1674–1681, 2000.
• Kaack etal. [2022]LynnH Kaack, PriyaL Donti, Emma Strubell, George Kamiya, Felix Creutzig, and David Rolnick.Aligning artificial intelligence with climate change mitigation.Nature Climate Change, 12(6):518–527, 2022.
• Rice etal. [2000]Peter Rice, Ian Longden, and Alan Bleasby.Emboss: the European molecular biology open software suite.Trends in genetics, 16(6):276–277, 2000.

Appendix A SUPPLEMENTARY MATERIAL

A.1 FIGURES