@inproceedings{liu-etal-2023-deplot,
    title = "{D}e{P}lot: One-shot visual language reasoning by plot-to-table translation",
    author = "Liu, Fangyu  and
      Eisenschlos, Julian  and
      Piccinno, Francesco  and
      Krichene, Syrine  and
      Pang, Chenxi  and
      Lee, Kenton  and
      Joshi, Mandar  and
      Chen, Wenhu  and
      Collier, Nigel  and
      Altun, Yasemin",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.660",
    doi = "10.18653/v1/2023.findings-acl.660",
    pages = "10381--10399",
    abstract = "Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models require at least tens of thousands of training examples and their reasoning capabilities are still much limited, especially on complex human-written queries. This paper presents the first one-shot solution to visual language reasoning. We decompose the challenge of visual language reasoning into two steps: (1) plot-to-text translation, and (2) reasoning over the translated text. The key in this method is a modality conversion module, named as DePlot, which translates the image of a plot or chart to a linearized table. The output of DePlot can then be directly used to prompt a pretrained large language model (LLM), exploiting the few-shot reasoning capabilities of LLMs. To obtain DePlot, we standardize the plot-to-table task by establishing unified task formats and metrics, and train DePlot end-to-end on this task. DePlot can then be used off-the-shelf together with LLMs in a plug-and-play fashion. Compared with a SOTA model finetuned on more than thousands of data points, DePlot+LLM with just one-shot prompting achieves a 29.4{\%} improvement over finetuned SOTA on human-written queries from the task of chart QA.",
}

Visual language such as charts and plots is ubiquitous in the human world. Comprehending plots and charts requires strong reasoning skills. Prior state-of-the-art (SOTA) models require at least tens of thousands of training examples and their reasoning capabilities are still much limited, especially on complex human-written queries. This paper presents the first one-shot solution to visual language reasoning. We decompose the challenge of visual language reasoning into two steps: (1) plot-to-text translation, and (2) reasoning over the translated text. The key in this method is a modality conversion module, named as DePlot, which translates the image of a plot or chart to a linearized table. The output of DePlot can then be directly used to prompt a pretrained large language model (LLM), exploiting the few-shot reasoning capabilities of LLMs. To obtain DePlot, we standardize the plot-to-table task by establishing unified task formats and metrics, and train DePlot end-to-end on this task. DePlot can then be used off-the-shelf together with LLMs in a plug-and-play fashion. Compared with a SOTA model finetuned on more than thousands of data points, DePlot+LLM with just one-shot prompting achieves a 29.4{\%} improvement over finetuned SOTA on human-written queries from the task of chart QA.
Liu, Fangyu Eisenschlos, Julian Piccinno, Francesco Krichene, Syrine Pang, Chenxi Lee, Kenton Joshi, M ar Chen, Wenhu Collier, Nigel Altun, Yasemin
DePlot: One-shot visual language reasoning by plot-to-table translation