Fig. 2: Improvement of phenotype prediction from cross-modal representations over unimodal representations or supervised learning from the original modalities.
From: Cross-modal autoencoder framework learns holistic representations of cardiovascular state
a A t-SNE visualization of the cross-modal embeddings for the ECG and MRI samples demonstrates that the modality specifc embeddings are well-mixed, unlike the modality specific embeddings obtained from the unimodal autoencoders. b Ranking each MRI by its cosine similarity with a given ECG in the latent space, we visualize the accuracy that the ground truth MRI appears in the top k neighbors among 4752 test ECG-MRI pairs from the UK Biobank. c Kernel regression on cross-modal representations outperforms kernel regression on unimodal representations and supervised deep learning methods on 4 different tasks: (1) prediction of ECG derived phenotypes from MRIs only (n = 4120, mean values are reported with error bars indicating one standard deviation); (2) prediction of MRI-derived phenotypes from ECG only (n = 4218, mean values are reported with error bars indicating one standard deviation); (3) prediction of general physiological phenotypes that are of categorical nature from either ECG or MRI (n = 4218, mean values are reported with error bars indicating one standard deviation); and (4) prediction of general physiological phenotypes that are of continuous nature from either ECG or MRI (n = 4212, mean values are reported with error bars indicating one standard deviation). All MRI phenotype abbreviations are defined in the “Methods” subsection “Models, data, and scaling law for phenotype prediction tasks”. Error bars are computed using 5-fold cross-validation. d Analysis of the scaling law when utilizing our framework for predicting MRI derived phenotypes from ECGs only. We observe that increasing the number of unlabelled ECG–MRI pairs for pre-training boosts the mean R2 prediction of 9 MRI-derived phenotypes by twice as much as increasing the number of labelled MRI samples. This analysis highlights the benefit of collecting more unlabelled ECG–MRI pairs as compared to paired labelled examples for this task.
