@inproceedings{hsieh-etal-2023-distilling,
    title = "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes",
    author = "Hsieh, Cheng-Yu  and
      Li, Chun-Liang  and
      Yeh, Chih-kuan  and
      Nakhost, Hootan  and
      Fujii, Yasuhisa  and
      Ratner, Alex  and
      Krishna, Ranjay  and
      Lee, Chen-Yu  and
      Pfister, Tomas",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.507",
    doi = "10.18653/v1/2023.findings-acl.507",
    pages = "8003--8017",
    abstract = "Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for training small models within a multi-task framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to few-shot prompted LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our finetuned 770M T5 model outperforms the few-shot prompted 540B PaLM model using only 80{\%} of available data on a benchmark, whereas standard finetuning the same T5 model struggles to match even by using 100{\%} of the dataset.",
}

Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for training small models within a multi-task framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to few-shot prompted LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our finetuned 770M T5 model outperforms the few-shot prompted 540B PaLM model using only 80{\%} of available data on a benchmark, whereas standard finetuning the same T5 model struggles to match even by using 100{\%} of the dataset.
Hsieh, Cheng-Yu Li, Chun-Liang Yeh, Chih-kuan Nakhost, Hootan Fujii, Yasuhisa Ratner, Alex Krishna, Ranjay Lee, Chen-Yu Pfister, Tomas
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes