@inproceedings{gonzalez-gutierrez-etal-2023-analyzing,
    title = "Analyzing Text Representations by Measuring Task Alignment",
    author = "Gonzalez-Gutierrez, Cesar  and
      Primadhanty, Audi  and
      Cazzaro, Francesco  and
      Quattoni, Ariadna",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-short.7",
    doi = "10.18653/v1/2023.acl-short.7",
    pages = "70--81",
    abstract = "Textual representations based on pre-trained language models are key, especially in few-shot learning scenarios. What makes a representation good for text classification? Is it due to the geometric properties of the space or because it is well aligned with the task? We hypothesize the second claim. To test it, we develop a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity. Our experiments on text classification validate our hypothesis by showing that task alignment can explain the classification performance of a given representation.",
}

Textual representations based on pre-trained language models are key, especially in few-shot learning scenarios. What makes a representation good for text classification? Is it due to the geometric properties of the space or because it is well aligned with the task? We hypothesize the second claim. To test it, we develop a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity. Our experiments on text classification validate our hypothesis by showing that task alignment can explain the classification performance of a given representation.
Gonzalez-Gutierrez, Cesar Primadhanty, Audi Cazzaro, Francesco Quattoni, Ariadna
Analyzing Text Representations by Measuring Task Alignment