Fig. 5: Efficient model architecture allows matching performance while strongly reducing the number of model parameters.
From: Nucleotide Transformer: building and evaluating robust foundation models for human genomics
a, NTr-v2 models loss value evolution during training as a function of the number of tokens seen so far. b, Normalized mean of MCC performance for NT-v2 models as a function of the number of tokens seen during pre-training. c, Normalized mean of MCC performance across downstream tasks (divided by category) for all NT (gray) and -v2 (blue shades) NT models after fine-tuning. Black dashed line represents the performance of the NT 2.5B Multispecies model. d, Overview of the NT fine-tuning on nucleotide-level splice site prediction task. Pre-trained weights and weights trained from scratch are highlighted. e, The Multispecies v2 500M model performance on predicting splice sites from the human genome, compared to its 2.5B counterpart and SpliceAI.
