@inproceedings{crisostomi-etal-2023-mitigating,
    title = "Mitigating the Burden of Redundant Datasets via Batch-Wise Unique Samples and Frequency-Aware Losses",
    author = "Crisostomi, Donato  and
      Caciolai, Andrea  and
      Pedrani, Alessandro  and
      Rottmann, Kay  and
      Manzotti, Alessandro  and
      Palumbo, Enrico  and
      Bernardi, Davide",
    editor = "Sitaram, Sunayana  and
      Beigman Klebanov, Beata  and
      Williams, Jason D",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-industry.23",
    doi = "10.18653/v1/2023.acl-industry.23",
    pages = "235--247",
    abstract = "Datasets used to train deep learning models in industrial settings often exhibit skewed distributions with some samples repeated a large number of times. This paper presents a simple yet effective solution to reduce the increased burden of repeated computation on redundant datasets. Our approach eliminates duplicates at the batch level, without altering the data distribution observed by the model, making it model-agnostic and easy to implement as a plug-and-play module. We also provide a mathematical expression to estimate the reduction in training time that our approach provides. Through empirical evidence, we show that our approach significantly reduces training times on various models across datasets with varying redundancy factors, without impacting their performance on the Named Entity Recognition task, both on publicly available datasets and in real industrial settings. In the latter, the approach speeds training by up to 87{\%}, and by 46{\%} on average, with a drop in model performance of 0.2{\%} relative at worst. We finally release a modular and reusable codebase to further advance research in this area.",
}

Datasets used to train deep learning models in industrial settings often exhibit skewed distributions with some samples repeated a large number of times. This paper presents a simple yet effective solution to reduce the increased burden of repeated computation on redundant datasets. Our approach eliminates duplicates at the batch level, without altering the data distribution observed by the model, making it model-agnostic and easy to implement as a plug-and-play module. We also provide a mathematical expression to estimate the reduction in training time that our approach provides. Through empirical evidence, we show that our approach significantly reduces training times on various models across datasets with varying redundancy factors, without impacting their performance on the Named Entity Recognition task, both on publicly available datasets and in real industrial settings. In the latter, the approach speeds training by up to 87{\%}, and by 46{\%} on average, with a drop in model performance of 0.2{\%} relative at worst. We finally release a modular and reusable codebase to further advance research in this area.
Crisostomi, Donato Caciolai, Andrea Pedrani, Aless ro Rottmann, Kay Manzotti, Aless ro Palumbo, Enrico Bernardi, Davide
Mitigating the Burden of Redundant Datasets via Batch-Wise Unique Samples and Frequency-Aware Losses