Fig. 3: Comparison of inference speed across model sizes and evaluation tasks.
From: Synthetic data distillation enables the extraction of clinical information at scale
a illustrates the average number of seconds needed to process an example for each dataset and model, b shows the average number of tokens read or ingested per second, and (c) depicts the average number of tokens generated per second. When comparing the center and right panels, note that token generation tends to be more time-consuming than token ingestion.
