@inproceedings{hueser-etal-2023-sharing,
    title = "Sharing Encoder Representations across Languages, Domains and Tasks in Large-Scale Spoken Language Understanding",
    author = "Hueser, Jonathan  and
      Gaspers, Judith  and
      Gueudre, Thomas  and
      Prakash, Chandana  and
      Cao, Jin  and
      Sorokin, Daniil  and
      Do, Quynh  and
      Anastassacos, Nicolas  and
      Falke, Tobias  and
      Gojayev, Turan",
    editor = "Sitaram, Sunayana  and
      Beigman Klebanov, Beata  and
      Williams, Jason D",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-industry.43",
    doi = "10.18653/v1/2023.acl-industry.43",
    pages = "447--456",
    abstract = "Leveraging representations from pre-trained transformer-based encoders achieves state-of-the-art performance on numerous NLP tasks. Larger encoders can improve accuracy for spoken language understanding (SLU) but are challenging to use given the inference latency constraints of online systems (especially on CPU machines).We evaluate using a larger 170M parameter BERT encoder that shares representations across languages, domains and tasks for SLU compared to using smaller 17M parameter BERT encoders with language-, domain- and task-decoupled finetuning.Running inference with a larger shared encoder on GPU is latency neutral and reduces infrastructure cost compared to running inference for decoupled smaller encoders on CPU machines. The larger shared encoder reduces semantic error rates by 4.62{\%} for test sets representing user requests to voice-controlled devices and 5.79{\%} on the tail of the test sets on average across four languages.",
}

Leveraging representations from pre-trained transformer-based encoders achieves state-of-the-art performance on numerous NLP tasks. Larger encoders can improve accuracy for spoken language understanding (SLU) but are challenging to use given the inference latency constraints of online systems (especially on CPU machines).We evaluate using a larger 170M parameter BERT encoder that shares representations across languages, domains and tasks for SLU compared to using smaller 17M parameter BERT encoders with language-, domain- and task-decoupled finetuning.Running inference with a larger shared encoder on GPU is latency neutral and reduces infrastructure cost compared to running inference for decoupled smaller encoders on CPU machines. The larger shared encoder reduces semantic error rates by 4.62{\%} for test sets representing user requests to voice-controlled devices and 5.79{\%} on the tail of the test sets on average across four languages.
Hueser, Jonathan Gaspers, Judith Gueudre, Thomas Prakash, Ch ana Cao, Jin Sorokin, Daniil Do, Quynh Anastassacos, Nicolas Falke, Tobias Gojayev, Turan
Sharing Encoder Representations across Languages, Domains and Tasks in Large-Scale Spoken Language Understanding