@inproceedings{zhao-etal-2023-robut,
    title = "{R}obu{T}: A Systematic Study of Table {QA} Robustness Against Human-Annotated Adversarial Perturbations",
    author = "Zhao, Yilun  and
      Zhao, Chen  and
      Nan, Linyong  and
      Qi, Zhenting  and
      Zhang, Wenlin  and
      Tang, Xiangru  and
      Mi, Boyu  and
      Radev, Dragomir",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.334",
    doi = "10.18653/v1/2023.acl-long.334",
    pages = "6064--6081",
    abstract = "Despite significant progress having been made in question answering on tabular data (Table QA), it{'}s unclear whether, and to what extent existing Table QA models are robust to task-specific perturbations, e.g., replacing key question entities or shuffling table columns. To systematically study the robustness of Table QA models, we propose a benchmark called RobuT, which builds upon existing Table QA datasets (WTQ, WikiSQL-Weak, and SQA) and includes human-annotated adversarial perturbations in terms of table header, table content, and question. Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets. We propose to address this problem by using large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models.",
}

Despite significant progress having been made in question answering on tabular data (Table QA), it{'}s unclear whether, and to what extent existing Table QA models are robust to task-specific perturbations, e.g., replacing key question entities or shuffling table columns. To systematically study the robustness of Table QA models, we propose a benchmark called RobuT, which builds upon existing Table QA datasets (WTQ, WikiSQL-Weak, and SQA) and includes human-annotated adversarial perturbations in terms of table header, table content, and question. Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets. We propose to address this problem by using large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models.
Zhao, Yilun Zhao, Chen Nan, Linyong Qi, Zhenting Zhang, Wenlin Tang, Xiangru Mi, Boyu Radev, Dragomir
RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations