What techniques have been proposed to perform table question answering using neural NLP models?
Table QA Question answering over tables has
received significant attention as it helps non-expert
users interact with complex tabular data. This prob-
lem is originally framed as semantic parsing, also
known as Text-to-SQL parsing (Yu et al., 2018,
2019; Wang et al., 2020b; Guo et al., 2021), in
which the parser takes both question and table
header as input, and predicts a SQL query that
is directly executable to get the answer. How-
ever, training state-of-the-art Text-to-SQL parsers
require large amounts of expensive SQL annota-
tions, limiting its applicability to real scenarios; In
addition, these Text-to-SQL parsers make a sim-
plified assumption that only table headers are nec-
essary while ignoring the value of table contents.
To mitigate these issues, recent works ignore gen-
erating SQL queries, and instead follow retrieve
then reason paradigm (Yin et al., 2020; Herzig
et al., 2020; Eisenschlos et al., 2020; Yang et al.,
2022; Liu et al., 2022; Jiang et al., 2022; Zhao
et al., 2022b), which first retrieve information from
the table, and conduct human-like reasoning to an-
swer the question. With the help of pre-training
on large scale table corpus, these approaches have
achieved remarkable results on several Table QA
benchmarks, including WikiTableQuestions (Pasu-
pat and Liang, 2015), WIKISQL-WEAK (Zhong
et al., 2017), and SQA (Iyyer et al., 2017). More re-
cently, Chen (2022) found that LLMs (Brown et al.,
2020; Chen et al., 2021) with few-shot in-context
learning shows promise on the Table QA task.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider.
[{"first":"Pengcheng","last":"Yin","middle":[]},{"first":"Graham","last":"Neubig","middle":[]},{"first":"Wen-tau","last":"Yih","middle":[]},{"first":"Sebastian","last":"Riedel","middle":[]}]
TaPas: Weakly Supervised Table Parsing via Pre-training
Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TaPas, an approach to question answering over tables without generating logical forms. TaPas trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TaPas extends BERT’s architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TaPas outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WikiSQL and WikiTQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WikiSQL to WikiTQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.
[{"first":"Jonathan","last":"Herzig","middle":[]},{"first":"Pawel Krzysztof","last":"Nowak","middle":[]},{"first":"Thomas","last":"Müller","middle":[]},{"first":"Francesco","last":"Piccinno","middle":[]},{"first":"Julian","last":"Eisenschlos","middle":[]}]
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Understanding tables is an important aspect of natural language understanding. Existing models for table understanding require linearization of the table structure, where row or column order is encoded as an unwanted bias. Such spurious biases make the model vulnerable to row and column order perturbations. Additionally, prior work has not thoroughly modeled the table structures or table-text alignments, hindering the table-text understanding ability. In this work, we propose a robust and structurally aware table-text encoding architecture TableFormer, where tabular structural biases are incorporated completely through learnable attention biases. TableFormer is (1) strictly invariant to row and column orders, and, (2) could understand tables better due to its tabular inductive biases. Our evaluations showed that TableFormer outperforms strong baselines in all settings on SQA, WTQ and TabFact table reasoning datasets, and achieves state-of-the-art performance on SQA, especially when facing answer-invariant row and column order perturbations (6% improvement over the best baseline), because previous SOTA models’ performance drops by 4% - 6% when facing such perturbations while TableFormer is not affected.
[{"first":"Jingfeng","last":"Yang","middle":[]},{"first":"Aditya","last":"Gupta","middle":[]},{"first":"Shyam","last":"Upadhyay","middle":[]},{"first":"Luheng","last":"He","middle":[]},{"first":"Rahul","last":"Goel","middle":[]},{"first":"Shachi","last":"Paul","middle":[]}]
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value. The intrinsic complexity of handling tables often adds an extra burden to both model design and data annotation. In this paper, we aim to develop a simple table-based QA model with minimal annotation effort. Motivated by the fact that table-based QA requires both alignment between questions and tables and the ability to perform complicated reasoning over multiple table elements, we propose an omnivorous pretraining approach that consumes both natural and synthetic data to endow models with these respective abilities. Specifically, given freely available tables, we leverage retrieval to pair them with relevant natural sentences for mask-based pretraining, and synthesize NL questions by converting SQL sampled from tables for pretraining with a QA loss. We perform extensive experiments in both few-shot and full settings, and the results clearly demonstrate the superiority of our model OmniTab, with the best multitasking approach achieving an absolute gain of 16.2% and 2.7% in 128-shot and full settings respectively, also establishing a new state-of-the-art on WikiTableQuestions. Detailed ablations and analyses reveal different characteristics of natural and synthetic data, shedding light on future directions in omnivorous pretraining.
[{"first":"Zhengbao","last":"Jiang","middle":[]},{"first":"Yi","last":"Mao","middle":[]},{"first":"Pengcheng","last":"He","middle":[]},{"first":"Graham","last":"Neubig","middle":[]},{"first":"Weizhu","last":"Chen","middle":[]}]
Understanding tables with intermediate pre-training
Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning. While there is extensive work on textual entailment, table entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to fine-tuning. This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task. To be able to use long examples as input of BERT models, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a moderate drop in accuracy. The different methods set the new state-of-the-art on the TabFact (Chen et al., 2020) and SQA datasets.
[{"first":"Julian","last":"Eisenschlos","middle":[]},{"first":"Syrine","last":"Krichene","middle":[]},{"first":"Thomas","last":"Müller","middle":[]}]
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes the improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks. Our code can be found at https://github.com/microsoft/Table-Pretraining.
[{"first":"Qian","last":"Liu","middle":[]},{"first":"Bei","last":"Chen","middle":[]},{"first":"Jiaqi","last":"Guo","middle":[]},{"first":"Morteza","last":"Ziyadi","middle":[]},{"first":"Zeqi","last":"Lin","middle":[]},{"first":"Weizhu","last":"Chen","middle":[]},{"first":"Jian-Guang","last":"Lou","middle":[]}]
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Reasoning over tabular data requires both table structure understanding and a broad set of table reasoning skills. Current models with table-specific architectures and pre-training methods perform well on understanding table structures, but they still struggle with tasks that require various table reasoning skills. In this work, we develop ReasTAP to show that high-level table reasoning skills can be injected into models during pre-training without a complex table-specific architecture design. We define 7 table reasoning skills, such as numerical operation, temporal comparison, and conjunction. Each reasoning skill is associated with one example generator, which synthesizes questions over semi-structured tables according to the sampled templates. We model the table pre-training task as a sequence generation task and pre-train ReasTAP to generate precise answers of the synthetic examples. ReasTAP is evaluated on four benchmarks covering three downstream tasks including 1) WikiSQL-Weak and WikiTQ for Table Question Answering, 2) TabFact for Table Fact Verification, and 3) LogicNLG for Faithful Table-to-Text Generation. Experimental results demonstrate that ReasTAP achieves new state-of-the-art results on all of them and delivers a significant improvement under low-resource setting. Our code is publicly available at https://github.com/Yale-LILY/ReasTAP.
[{"first":"Yilun","last":"Zhao","middle":[]},{"first":"Linyong","last":"Nan","middle":[]},{"first":"Zhenting","last":"Qi","middle":[]},{"first":"Rui","last":"Zhang","middle":[]},{"first":"Dragomir","last":"Radev","middle":[]}]
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Recent years have witnessed a rapid advance in the ability to understand and answer questions about free-form natural language (NL) text (Rajpurkar et al., 2016), largely due to large-scale, pretrained language models (LMs) like BERT (Devlin et al., 2019). These models allow us to capture the syntax and semantics of text via representations learned in an unsupervised manner, before fine-tuning the model to downstream tasks (Melamud et al., 2016;McCann et al., 2017;Peters et al., 2018;Goldberg, 2019). It is also relatively easy to apply such pretrained LMs to comprehension tasks that are modeled as text span selection problems, where the boundary of an answer span can be predicted using a simple classifier on top of the LM .
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
However, it is less clear how one could pretrain and fine-tune such models for other QA tasks that involve joint reasoning over both free-form NL text and structured data. One example task is semantic parsing for access to databases (DBs) (Zelle and Mooney, 1996;Berant et al., 2013;Yih et al., 2015), the task of transducing an NL utterance (e.g., "Which country has the largest GDP?") into a structured query over DB tables (e.g., SQL querying a database of economics). A key challenge in this scenario is understanding the structured schema of DB tables (e.g., the name, data type, and stored values of columns), and more importantly, the alignment between the input text and the schema (e.g., the token "GDP" refers to the Gross Domestic Product column), which is essential for inferring the correct DB query (Berant and Liang, 2014).
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Neural semantic parsers tailored to this task therefore attempt to learn joint representations of NL utterances and the (semi-)structured schema of DB tables (e.g., representations of its columns or cell values, as in Krishnamurthy et al. (2017); Bogin et al. (2019b); Wang et al. (2019a), inter alia). However, this unique setting poses several challenges in applying pretrained LMs. First, information stored in DB tables exhibit strong underlying structure, while existing LMs (e.g., BERT) are solely trained for encoding free-form text. Second, a DB table could potentially have a large number of rows, and naively encoding all of them using a resource-heavy LM is computationally intractable. Finally, unlike most text-based QA tasks (e.g., SQuAD, Rajpurkar et al. (2016)) which could be formulated as a generic answer span selection problem and solved by a pretrained model with additional classification layers, semantic parsing is highly domain-specific, and the architecture of a neural parser is strongly coupled with the structure of its underlying DB (e.g., systems for SQL-based and other types of DBs use different encoder mod-els). In fact, existing systems have attempted to leverage BERT, but each with their own domainspecific, in-house strategies to encode the structured information in the DB (Guo et al., 2019;Zhang et al., 2019a;Hwang et al., 2019), and importantly, without pretraining representations on structured data. These challenges call for development of general-purpose pretraining approaches tailored to learning representations for both NL utterances and structured DB tables.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
In this paper we present TABERT, a pretraining approach for joint understanding of NL text and (semi-)structured tabular data ( § 3). TABERT is built on top of BERT, and jointly learns contextual representations for utterances and the structured schema of DB tables (e.g., a vector for each utterance token and table column). Specifically, TABERT linearizes the structure of tables to be compatible with a Transformer-based BERT model. To cope with large tables, we propose content snapshots, a method to encode a subset of table content most relevant to the input utterance. This strategy is further combined with a vertical attention mechanism to share information among cell representations in different rows ( § 3.1). To capture the association between tabular data and related NL text, TABERT is pretrained on a parallel corpus of 26 million tables and English paragraphs ( § 3.2).
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
TABERT can be plugged into a neural semantic parser as a general-purpose encoder to compute representations for utterances and tables. Our key insight is that although semantic parsers are highly domain-specific, most systems rely on representations of input utterances and the table schemas to facilitate subsequent generation of DB queries, and these representations can be provided by TABERT, regardless of the domain of the parsing task.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
We apply TABERT to two different semantic parsing paradigms: (1) a classical supervised learning setting on the SPIDER text-to-SQL dataset (Yu et al., 2018c), where TABERT is fine-tuned together with a task-specific parser using parallel NL utterances and labeled DB queries ( § 4.1); and (2) a challenging weakly-supervised learning benchmark WIKITABLEQUESTIONS (Pasupat and Liang, 2015), where a system has to infer latent DB queries from its execution results ( § 4.2). We demonstrate TABERT is effective in both scenarios, showing that it is a drop-in replacement of a parser's original encoder for computing contextual representations of NL utterances and DB tables.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Specifically, systems augmented with TABERT outperforms their counterparts using BERT, registering state-of-the-art performance on WIKITABLE-QUESTIONS, while performing competitively on SPIDER ( § 5).
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Semantic Parsing over Tables Semantic parsing tackles the task of translating an NL utterance u into a formal meaning representation (MR) z. Specifically, we focus on parsing utterances to access database tables, where z is a structured query (e.g., an SQL query) executable on a set of rela-
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
, one for each column c j . Each cell s i,j contains a list of tokens.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Depending on the underlying data representation schema used by the DB, a table could either be fully structured with strongly-typed and normalized contents (e.g., a table column named distance has a unit of kilometers, with all of its cell values, like 200, bearing the same unit), as is commonly the case for SQL-based DBs ( § 4.1). Alternatively, it could be semi-structured with unnormalized, textual cell values (e.g., 200 km, § 4.2). The query language could also take a variety of forms, from general-purpose DB access languages like SQL to domain-specific ones tailored to a particular task.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Given an utterance and its associated tables, a neural semantic parser generates a DB query from the vector representations of the utterance tokens and the structured schema of tables. In this paper we refer schema as the set of columns in a table, and its representation as the list of vectors that represent its columns 2 . We will introduce how TABERT computes these representations in § 3.1.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Masked Language Models Given a sequence of NL tokens x = x 1 , x 2 , . . . , x n , a masked language model (e.g., BERT) is an LM trained using the masked language modeling objective, which aims to recover the original tokens in x from a "corrupted" context created by randomly masking out certain tokens in x. Specifically, let x m = {x i 1 , . . . , x im } be the subset of tokens in x selected to be masked out, and x denote the masked sequence with tokens in x m replaced by a  tion p θ (x m | x) over the target tokens x m given the masked context x. BERT parameterizes p θ (x m | x) using a Transformer model. During the pretraining phase, BERT maximizes p θ (x m | x) on large-scale textual corpora. In the fine-tuning phase, the pretrained model is used as an encoder to compute representations of input NL tokens, and its parameters are jointly tuned with other task-specific neural components.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
We first present how TABERT computes representations for NL utterances and table schemas ( § 3.1), and then describe the pretraining procedure ( § 3.2).
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Fig. 1 presents a schematic overview of TABERT. Given an utterance u and a table T , TABERT first creates a content snapshot of T . This snapshot consists of sampled rows that summarize the information in T most relevant to the input utterance. The model then linearizes each row in the snapshot, concatenates each linearized row with the utterance, and uses the concatenated string as input to a Transformer (e.g., BERT) model, which outputs row-wise encoding vectors of utterance tokens and cells. The encodings for all the rows in 3 Example adapted from stanford.io/38iZ8Pf the snapshot are fed into a series of vertical selfattention layers, where a cell representation (or an utterance token representation) is computed by attending to vertically-aligned vectors of the same column (or the same NL token). Finally, representations for each utterance token and column are generated from a pooling layer.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
One major feature of TABERT is its use of the table contents, as opposed to just using the column names, in encoding the table schema. This is motivated by the fact that contents provide more detail about the semantics of a column than just the column's name, which might be ambiguous. For instance, the Venue column in Fig. 1 which is used to answer the example question actually refers to host cities, and encoding the sampled cell values while creating its representation may help match the term "city" in the input utterance to this column. However, a DB table could potentially have a large number of rows, with only few of them actually relevant to answering the input utterance. Encoding all of the contents using a resource-heavy Transformer is both computationally intractable and likely not necessary. Thus, we instead use a content snapshot consisting of only a few rows that are most relevant to the input utterance, providing an efficient approach to calculate content-sensitive column representations from cell values.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
We use a simple strategy to create content snap-shots of K rows based on the relevance between the utterance and a row. For K > 1, we select the top-K rows in the input table that have the highest n-gram overlap ratio with the utterance. 4 For K = 1, to include in the snapshot as much information relevant to the utterance as possible, we create a synthetic row by selecting the cell values from each column that have the highest n-gram overlap with the utterance. Using synthetic rows in this restricted setting is motivated by the fact that cell values most relevant to answer the utterance could come from multiple rows. As an example, consider the utterance "How many more participants were there in 2008 than in the London Olympics?", and an associating table with columns Year, Host City and Number of Participants, the most relevant cells to the utterance, 2008 (from Year) and London (from Host City), are from different rows, which could be included in a single synthetic row. In the initial experiments we found synthetic rows also help stabilize learning.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Row Linearization TABERT creates a linearized sequence for each row in the content snapshot as input to the Transformer model. Fig. 1(B) depicts the linearization for R 2 , which consists of a concatenation of the utterance, columns, and their cell values. Specifically, each cell is represented by the name and data type 5 of the column, together with its actual value, separated by a vertical bar. As an example, the cell s 2,1 valued 2005 in R 2 in Fig. 1 is encoded as
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Year Column Name
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Column Type
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Cell Value
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
(1)
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
The linearization of a row is then formed by concatenating the above string encodings of all the cells, separated by the [SEP] symbol. We then prefix the row linearization with utterance tokens as input sequence to the Transformer.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Existing works have applied different linearization strategies to encode tables with Transformers (Hwang et al., 2019;, while our row approach is specifically designed for encoding content snapshots. We present in § 5 results with different linearization choices.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Vertical Self-Attention Mechanism The base Transformer model in TABERT outputs vector encodings of utterance and cell tokens for each row. These row-level vectors are computed separately and therefore independent of each other. To allow for information flow across cell representations of different rows, we propose vertical self-attention, a self-attention mechanism that operates over vertically aligned vectors from different rows.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
As in Fig. 1(C), TABERT has V stacked verticallevel self-attention layers. To generate aligned inputs for vertical attention, we first compute a fixedlength initial vector for each cell at position i, j , which is given by mean-pooling over the sequence of the Transformer's output vectors that correspond to its variable-length linearization as in Eq. (1). Next, the sequence of word vectors for the NL utterance (from the base Transformer model) are concatenated with the cell vectors as initial inputs to the vertical attention layer.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Each vertical attention layer has the same parameterization as the Transformer layer in (Vaswani et al., 2017), but operates on vertically aligned elements, i.e., utterance and cell vectors that correspond to the same question token and column, respectively. This vertical self-attention mechanism enables the model to aggregate information from different rows in the content snapshot, allowing TABERT to capture cross-row dependencies on cell values.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Utterance and Column Representations A representation c j is computed for each column c j by mean-pooling over its vertically aligned cell vectors, {s i,j : R i in content snapshot}, from the last vertical layer. A representation for each utterance token, x j , is computed similarly over the vertically aligned token vectors. These representations will be used by downstream neural semantic parsers. TABERT also outputs an optional fixedlength table representation T using the representation of the prefixed [CLS] symbol, which is useful for parsers that operate on multiple DB tables.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Training Data Since there is no large-scale, high-quality parallel corpus of NL text and structured tables, we instead use semi-structured tables that commonly exist on the Web as a surrogate data source. As a first step in this line, we focus on collecting parallel data in English, while extending to multilingual scenarios would be an interesting avenue for future work. Specifically, we collect tables and their surrounding NL text from English Wikipedia and the WDC WebTable Corpus (Lehmberg et al., 2016), a large-scale table collection from CommonCrawl. The raw data is extremely noisy, and we apply aggressive cleaning heuristics to filter out invalid examples (e.g., examples with HTML snippets or in foreign languages, and non-relational tables without headers). See Appendix § A.1 for details of data pre-processing. The pre-processed corpus contains 26.6 million parallel examples of tables and NL sentences. We perform sub-tokenization using the Wordpiece tokenizer shipped with BERT.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Unsupervised Learning Objectives We apply different objectives for learning representations of the NL context and structured tables. For NL contexts, we use the standard Masked Language Modeling (MLM) objective (Devlin et al., 2019), with a masking rate of 15% sub-tokens in an NL context.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
For learning column representations, we design two objectives motivated by the intuition that a column representation should contain both the general information of the column (e.g., its name and data type), and representative cell values relevant to the NL context. First, a Masked Column Prediction (MCP) objective encourages the model to recover the names and data types of masked columns. Specifically, we randomly select 20% of the columns in an input table, masking their names and data types in each row linearization (e.g., if the column Year in Fig. 1 is selected, the tokens Year and real in Eq. (1) will be masked). Given the column representation c j , TABERT is trained to predict the bag of masked (name and type) tokens from c j using a multi-label classification objective. Intuitively, MCP encourages the model to recover column information from its contexts.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Next, we use an auxiliary Cell Value Recovery (CVR) objective to ensure information of representative cell values in content snapshots is retained after additional layers of vertical self-attention. Specifically, for each masked column c j in the above MCP objective, CVR predicts the original tokens of each cell s i,j (of c j ) in the content snapshot conditioned on its cell vector s i,j . 6 For instance, for the example cell s 2,1 in Eq. (1), we predict its value 2005 from s 2,1 . Since a cell could have multiple value tokens, we apply the span-based prediction objective . Specifically, to predict a cell token s i,j k ∈ s i,j , its positional embedding e k and the cell representations s i,j are fed into a two-layer network f (·) with GeLU activations (Hendrycks and Gimpel, 2016). The output of f (·) is then used to predict the original value token s i,j k from a softmax layer.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Parsing over Tables
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
We apply TABERT for representation learning on two semantic parsing paradigms, a classical supervised text-to-SQL task over structured DBs ( § 4.1), and a weakly supervised parsing problem on semistructured Web tables ( § 4.2).
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Benchmark Dataset Supervised learning is the typical scenario of learning a parser using parallel data of utterances and queries. We use SPIDER (Yu et al., 2018c), a text-to-SQL dataset with 10,181 examples across 200 DBs. Each example consists of an utterance (e.g., "What is the total number of languages used in Aruba?"), a DB with one or more tables, and an annotated SQL query, which typically involves joining multiple tables to get the answer (e.g., SELECT COUNT(*) FROM Country JOIN Lang ON Country.Code = Lang.CountryCode WHERE Name = 'Aruba').
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Base Semantic Parser We aim to show TABERT could help improve upon an already strong parser. Unfortunately, at the time of writing, none of the top systems on SPIDER were publicly available. To establish a reasonable testbed, we developed our in-house system based on TranX (Yin and Neubig, 2018), an open-source general-purpose semantic parser. TranX translates an NL utterance into an intermediate meaning representation guided by a user-defined grammar. The generated intermediate MR could then be deterministically converted to domain-specific query languages (e.g., SQL). We use TABERT as encoder of utterances and table schemas. Specifically, for a given utterance u and a DB with a set of tables T = {T t }, we first pair u with each table T t in T as inputs to TABERT, which generates |T | sets of table-specific representations of utterances and columns. At each time step, an LSTM decoder performs hierarchical attention (Libovický and Helcl, 2017) over the list of table-specific representations, constructing an MR based on the predefined grammar. Following the IRNet model (Guo et al., 2019) which achieved the best performance on SPIDER as the time of writing, we use SemQL, a simplified version of the SQL, as the underlying grammar. We refer interested readers to Appendix § B.1 for details of our system.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Benchmark Dataset Weakly supervised semantic parsing considers the reinforcement learning task of inferring the correct query from its execution results (i.e., whether the answer is correct). Compared to supervised learning, weakly supervised parsing is significantly more challenging, as the parser does not have access to the labeled query, and has to explore the exponentially large search space of possible queries guided by the noisy binary reward signal of execution results.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
WIKITABLEQUESTIONS (Pasupat and Liang, 2015) is a popular dataset for weakly supervised semantic parsing, which has 22,033 utterances and 2,108 semi-structured Web tables from Wikipedia. 7 Compared to SPIDER, examples in this dataset do not involve joining multiple tables, but typically require compositional, multi-hop reasoning over a series of entries in the given table (e.g., to answer the example in Fig. 1 the parser needs to reason over the row set {R 2 , R 3 , R 5 }, locating the Venue field with the largest value of Year).
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Base Semantic Parser MAPO (Liang et al., 2018) is a strong system for weakly supervised semantic parsing. It improves the sample efficiency of the REINFORCE algorithm by biasing the exploration of queries towards the high-rewarding ones already discovered by the model. MAPO uses a domain-specific query language tailored to answering compositional questions on single tables, and its utterances and column representations are derived from an LSTM encoder, which we replaced with our TABERT model. See Appendix § B.2 for details of MAPO and our adaptation.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
In this section we evaluate TABERT on downstream tasks of semantic parsing to DB tables.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
We train two variants of the model, TABERT Base and TABERT Large , with the underlying Transformer model initialized with the uncased versions of BERT Base and BERT Large , respectively. 8 During pretraining, for each table and its associated NL context in the corpus, we create a series of training instances of paired NL sentences (as synthetically generated utterances) and tables (as content snapshots) by (1) sliding a (non-overlapping) context window of sentences with a maximum length of 128 tokens, and (2) using the NL tokens in the window as the utterance, and pairing it with randomly sampled rows from the table as content snapshots. TABERT is implemented in PyTorch using distributed training. Refer to Appendix § A.2 for details of pretraining.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Comparing Models We mainly present results for two variants of TABERT by varying the size of content snapshots K. TABERT(K = 3) uses three rows from input tables as content snapshots and three vertical self-attention layers. TABERT(K = 1) uses one synthetically generated row as the content snapshot as described in § 3.1. Since this model does not have multi-row input, we do not use additional vertical attention layers (and the cell value recovery learning objective). Its column representation c j is defined by mean-pooling over the Transformer's output encodings that correspond to the column name (e.g., the representation for the Year column in Fig. 1 is derived from the vector of the Year token in Eq. (1)). We find this strategy gives better results compared with using the cell representation s j as c j . We also compare with BERT using the same row linearization and content snapshot approach as TABERT(K = 1), which reduces to a TABERT(K = 1) model without pretraining on tabular corpora.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Evaluation Metrics As standard, we report execution accuracy on WIKITABLEQUESTIONS and exact-match accuracy of DB queries on SPIDER.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Tab. 1 and Tab. 2 summarize the end-to-end evaluation results on WIKITABLEQUESTIONS and SPI-DER, respectively. First, comparing with existing strong semantic parsing systems, we found our On SPIDER, our semantic parser based on TranX and SemQL ( § 4.1) is conceptually similar to the base version of IRNet as both systems use the SemQL grammar, while our system has a simpler decoder. Interestingly, we observe that its performance with BERT Base (61.8%) matches the full BERT-augmented IRNet model with a stronger decoder using augmented memory and coarse-to-fine decoding (61.9%). This confirms that our base parser is an effective baseline. Augmented with representations produced by TABERT Large (K = 3), our parser achieves up to 65.2% exact-match accuracy, a 2.8% increase over the base model using BERT Base . Note that while other competitive systems on the leaderboard use BERT with more sophisticated semantic parsing models, our best DEV. result is already close to the score registered by the best submission (RyanSQL+BERT). This suggests that if they instead used TABERT as the representation layer, they would see further gains.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Comparing semantic parsers augmented with Top-ranked Systems on Spider Leaderboard Model DEV. ACC. Global-GNN (Bogin et al., 2019a) 52.7 EditSQL + BERT (Zhang et al., 2019a) 57.6 RatSQL (Wang et al., 2019a) 60.9 IRNet + BERT (Guo et al., 2019) 60.3 + Memory + Coarse-to-Fine 61.9 IRNet V2 + BERT 63.9 RyanSQL + BERT (Choi et al., 2020) 66.6 Our System based on TranX (Yin and Neubig, 2018) Mean  TABERT and BERT, we found TABERT is more effective across the board. We hypothesize that the performance improvements would be attributed by two factors. First, pre-training on large parallel textual and tabular corpora helps TABERT learn to encode structure-rich tabular inputs in their linearized form (Eq. (1)), whose format is different from the ordinary natural language data that BERT is trained on. Second, pre-training on parallel data could also helps the model produce representations that better capture the alignment between an utterance and the relevant information presented in the structured schema, which is important for semantic parsing.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Overall, the results on the two benchmarks demonstrate that pretraining on aligned textual and tabular data is necessary for joint understanding of NL utterances and tables, and TABERT works well with both structured (SPIDER) and semi-structured (WIKITABLEQUESTIONS) DBs, and agnostic of the task-specific structures of semantic parsers.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
In this paper we propose using content snapshots to capture the information in input DB tables that is most relevant to answering the NL utterance. We therefore study the effectiveness of including content snapshots when generating schema representations. We include in Tab. 1 and Tab. 2 results of models without using content in row linearization ("−content snapshot"). Under this setting a column is rep-u: How many years before was the film Bacchae out before the Watermelon?  resented as "Column Name | Type" without cell values (c.f., Eq. (1)). We find that content snapshots are helpful for both BERT and TABERT, especially for TABERT. As discussed in § 3.1, encoding sampled values from columns in learning their representations helps the model infer alignments between entity and relational phrases in the utterance and the corresponding column. This is particularly helpful for identifying relevant columns from a DB table that is mentioned in the input utterance. As an example, empirically we observe that on SPIDER our semantic parser with TABERT Base using just one row of content snapshots (K = 1) registers a higher accuracy of selecting the correct columns when generating SQL queries (e.g., columns in SELECT and WHERE clauses), compared to the TABERT Base model without encoding content information (87.4% v.s. 86.4%). Additionally, comparing TABERT using one synthetic row (K = 1) and three rows from input tables (K = 3) as content snapshots, the latter generally performs better. Intuitively, encoding more table contents relevant to the input utterance could potentially help answer questions that involve reasoning over information across multiple rows in the table. Tab. 3 shows such an example, and to answer this question a parser need to subtract the values of Year in the rows for "The Watermelon" and "The Bacchae". TABERT Large (K = 3) is able to capture the two target rows in its content snapshot and generates the correct DB query, while the TABERT Large (K = 1) model with only one row as content snapshot fails to answer this example.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Effect of Row Linearization TABERT uses row linearization to represent a table row as sequential input to Transformer. Tab. 4 (upper half) presents results using various linearization methods. We find adding type information and content snapshots improves performance, as they provide more hints about the meaning of a column.   We also compare with existing linearization methods in literature using a TABERT Base model, with results shown in Tab. 4 (lower half). Hwang et al. (2019) uses BERT to encode concatenated column names to learn column representations. In line with our previous discussion on the effectiveness content snapshots, this simple strategy without encoding cell contents underperforms (although with TABERT Base pretrained on our tabular corpus the results become slightly better). Additionally, we remark that linearizing table contents has also be applied to other BERT-based tabular reasoning tasks. For instance,  propose a "natural" linearization approach for checking if an NL statement entails the factual information listed in a table using a binary classifier with representations from BERT, where a table is linearized by concatenating the semicolon-separated cell linearization for all rows. Each cell is represented by a phrase "column name is cell value". For completeness, we also tested this cell linearization approach, and find BERT Base achieved improved results. We leave pretraining TABERT with this linearization strategy as promising future work.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Impact of Pretraining Objectives TABERT uses two objectives ( § 3.2), a masked column prediction (MCP) and a cell value recovery (CVR) objective, to learn column representations that could capture both the general information of the column (via MCP) and its representative cell values related to the utterance (via CVR). Tab. 5 shows ablation results of pretraining TABERT with different objectives. We find TABERT trained with both MCP and the auxiliary CVR objectives gets a slight advantage, suggesting CVR could potentially lead to more representative column representations with additional cell information.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Semantic Parsing over Tables Tables are impor-tant media of world knowledge. Semantic parsers have been adapted to operate over structured DB tables (Wang et al., 2015;Xu et al., 2017;Dong and Lapata, 2018;Yu et al., 2018b;Shi et al., 2018;Wang et al., 2018), and open-domain, semistructured Web tables (Pasupat and Liang, 2015;Sun et al., 2016;Neelakantan et al., 2016). To improve representations of utterances and tables for neural semantic parsing, existing systems have applied pretrained word embeddings (e.g.., GloVe, as in Zhong et al. (2017); Yu et al. (2018a);Sun et al. (2018); Liang et al. (2018)), and BERT-family models for learning joint contextual representations of utterances and tables, but with domain-specific approaches to encode the structured information in tables (Hwang et al., 2019;He et al., 2019;Guo et al., 2019;Zhang et al., 2019a). TABERT advances this line of research by presenting a generalpurpose, pretrained encoder over parallel corpora of Web tables and NL context. Another relevant direction is to augment representations of columns from an individual table with global information of its linked tables defined by the DB schema (Bogin et al., 2019a;Wang et al., 2019a). TABERT could also potentially improve performance of these systems with improved table-level representations.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Knowledge-enhanced Pretraining Recent pretraining models have incorporated structured information from knowledge bases (KBs) or other structured semantic annotations into training contextual word representations, either by fusing vector representations of entities and relations on KBs into word representations of LMs (Peters et al., 2019;Zhang et al., 2019b,c), or by encouraging the LM to recover KB entities and relations from text (Sun et al., 2019;Liu et al., 2019a). TABERT is broadly relevant to this line in that it also exposes an LM with structured data (i.e., tables), while aiming to learn joint representations for both textual and structured tabular data.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
We present TABERT, a pretrained encoder for joint understanding of textual and tabular data. We show that semantic parsers using TABERT as a general-purpose feature representation layer achieved strong results on two benchmarks. This work also opens up several avenues for future work. First, we plan to evaluate TABERT on other related tasks involving joint reasoning over textual and tabular data (e.g. , table retrieval and table-totext generation). Second, following the discussions in § 5, we will explore other table linearization strategies with Transformers, improving the quality of pretraining corpora, as well as novel unsupervised objectives. Finally, to extend TABERT to cross-lingual settings with utterances in foreign languages and structured schemas defined in English, we plan to apply more advanced semantic similarity metrics for creating content snapshots.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
We collect parallel examples of tables and their surrounding NL sentences from two sources:
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Wikipedia Tables We extract all the tables on English Wikipedia 9 . For each table, we use the preceding three paragraphs as the NL context, as we observe that most Wiki tables are located after where they are described in the body text. (Lehmberg et al., 2016) is a large collection of Web tables extracted from the Common Crawl Web scrape 10 . We use its 2015 English-language relational subset, which consists of 50.8 million relational tables and their surrounding NL contexts.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Preprocessing Our dataset is collected from arbitrary Web tables, which are extremely noisy. We develop a set of heuristics to clean the data by: (1) removing columns whose names have more than 10 tokens; (2) filtering cells with more than two non-ASCII characters or 20 tokens; (3) removing empty or repetitive rows and columns; (4) filtering tables with less than three rows and four columns, and (5) running spaCy to identify the data type of columns (text or real value) by majority voting over the NER labels of column tokens, (6) rotating vertically oriented tables. We sub-tokenize the corpus using the Wordpiece tokenizer in Devlin et al. (2019). The pre-processing results in 1.3 million tables from Wikipedia and 25.3 million tables from the WDC corpus.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
As discussed in § 5, we create training instances of NL sentences (as synthetic utterances) and content snapshots from tables by sampling from the parallel corpus of NL contexts and tables. Each epoch contains 37.6M training instances. We train TABERT for 10 epochs. Tab. 6 lists the hyper-parameters used in training. Learning rates are validated on the development set of WIKITABLEQUESTIONS. We use a batch size of 512 for large models to reduce training time. The training objective is sum of the three pretraining objectives in § 3.2: the masked language modeling (MLM) objective for utterance tokens, the masked column prediction (MCP) objective for columns, and the column value recovery (CVR) objective for their cell values. An exception is pretraining the TABERT(K = 1) models. Since there are no additional vertical attention layers, we do not use the CVR objective, and the MCP objective reduces to the vanilla MLM objective over encodings from the base Transformer model. Our largest model TABERT Large (K = 3) takes six days to train for 10 epochs on 128 Tesla V100 GPUs using mixed precision training.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
B.1 Supervised Parsing on SPIDER Model We develop our text-to-SQL parser based on TranX (Yin and Neubig, 2018), which translates an NL utterance into a tree-structured abstract meaning representation following user-specified grammar, before deterministically convert the generated abstract MR into an SQL query. TranX models the construction process of an abstract MR (treestructured representation of an SQL query) using a transition-based system, which decomposes its generation story into a sequence of actions following the user defined grammar.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Formally, given an input NL utterance u and a database with a set of tables T = {T i }, the probability of generating of an SQL query (i.e., its semantically equivalent MR) z is decomposed as the production of action probabilities:
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
where a t is the action applied to the hypothesis at time stamp t. a <t denote the previous action history. We refer readers to Yin and Neubig (2018) for details of the transition system and how individual action probabilities are computed. In our adaptation of TranX to text-to-SQL parsing on SPIDER, we follow Guo et al. (2019) and use SemQL as the underlying grammar, which is a simplification of the SQL language. Fig. 2 lists the SemSQL grammar specified using the abstract syntax description language (Wang et al., 1997). Intuitively, the generation starts from a treestructured derivation with the root production rule select stmt →SelectStatement, which lays out overall the structure of an SQL query. At each time step, the decoder algorithm locates the current opening node on the derivation tree, following a depth-first, left-to-right order. If the opening node Parameter TABERT Base (K = 1) TABERT Large (K = 1) TABERT Base (K = 3) TABERT Large (K = 3)  is not a leaf node, the decoder invokes an action a t which expands the opening node using a production rule with appropriate type. If the current opening node is a leaf node (e.g., a node denoting string literal), the decoder fills in the leaf node using actions that emit terminal values.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
To use such a transition system to generate SQL queries, we extend its action space with two new types of actions, SELECTTABLE(T i ) for node of type table ref in Fig. 2, which selects a table T i (e.g., for predicting target tables for a FROM clause), and SELECTCOLUMN(T i , c j ) for node of type column ref, which selects the column c j from table T i (e.g., for predicting a result column used in the SELECT clause).
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
As described in § 4.1, TABERT produces a list of entries, with one entry T i , X i , C i for each table T i : M = T i , X i = {x 1 , x 2 , . . .},
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
where each entry T i , X i , C i in M consists of T i , the representation of table T i given by the output vector of the prefixed [CLS] symbol, the tablespecific representations of utterance tokens X i = {x 1 , x 2 , . . .}, and representations of columns in T i , C i = {c 1 , c 2 , . . .}. At each time step t, the decoder in TranX performs hierarchical attention over representations in M to compute a context vector. First, a table-wise attention score is computed using the LSTM's previous state, state t−1 with the set of table representations. score(T i ) = Softmax DotProduct(state t−1 , key(T i )) , (4) where the linear projection key(·) ∈ R 256 projects the table representations to key space. Next, for each table T i ∈ T , a table-wise context vector ctx(T i ) is generated by attending over the union of vectors in utterance token representations X i and column representations C i : ctx(T i ) = DotProductAttention state t−1 , key(X i ∪ C i ), value(X i ∪ C i ) , (5) with the LSTM state as the query, key(·) as the key, and another linear transformation value(·) ∈ R 256 to project the representations to value vectors. The final context vector is then given by the weighted sum of these table-wise context vectors ctx(T i ) (i ∈ {1, . . . , |T |}) weighted by the attention scores score(T i ). The generated context vector is then used to update the state of the decoder LSTM to state t .
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
The updated decoder state is then used to compute the probability of carrying out the action defined at time step t, a t . For a SELECTTABLE(T i ) action, its probability of is defined similarly as Eq. (4). For a SELECTCOLUMN(T i , c j ) action, it is factorized as the probability of selecting the table T i (given by Eq. (4)), times the probability of selecting the column c j . The latter is defined as score(c j ) = Softmax DotProduct(state t , c j ) .
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
(6) We also add simple entity linking features to the representations in M, defined by the following heuristics: (1) If an utterance token x ∈ u matches with the name of a table T , we concatenate a trainable embedding vector (table match ∈ R 16 ) to the representations of x and T . (2) Similarly, we concatenate an embedding vector (column match ∈ R 16 ) to the representations of an utterance token and a column if their names match.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
(3) Finally, we concatenate a zero-vector (0 ∈ R 16 ) to representations of all unmatched elements.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Configuration We use the default configuration of TranX. For TABERT parameters, we use an Adam optimizer with a learning rate of 3e − 5 and linearly decayed learning rate schedule, and
TaPas: Weakly Supervised Table Parsing via Pre-training
Question answering from semi-structured tables is usually seen as a semantic parsing task where the question is translated to a logical form that can be executed against the table to retrieve the correct denotation (Pasupat and Liang, 2015;Zhong et al., 2017;Agarwal et al., 2019). Semantic parsers rely on supervised training data that pairs natural language questions with logical forms, but such data is expensive to annotate.
TaPas: Weakly Supervised Table Parsing via Pre-training
In recent years, many attempts aim to reduce the burden of data collection for semantic parsing, including paraphrasing (Wang et al., 2015), human in the loop (Iyer et al., 2017;Lawrence and Riezler, 2018) and training on examples from other domains (Herzig and Berant, 2017;Su and Yan, 2017). One prominent data collection approach focuses on weak supervision where a training example consists of a question and its denotation instead of the full logical form (Clarke et al., 2010;Liang et al., 2011;Artzi and Zettlemoyer, 2013). Although appealing, training semantic parsers from this input is often difficult due to the abundance of spurious logical forms (Berant et al., 2013;Guu et al., 2017) and reward sparsity (Agarwal et al., 2019;Muhlgay et al., 2019). In addition, semantic parsing applications only utilize the generated logical form as an intermediate step in retrieving the answer. Generating logical forms, however, introduces difficulties such as maintaining a logical formalism with sufficient expressivity, obeying decoding constraints (e.g. wellformedness), and the label bias problem (Andor et al., 2016;Lafferty et al., 2001).
TaPas: Weakly Supervised Table Parsing via Pre-training
In this paper we present TAPAS (for Table  Parser), a weakly supervised question answering model that reasons over tables without generating logical forms. TAPAS predicts a minimal program by selecting a subset of the table cells and a possible aggregation operation to be executed on top of them. Consequently, TAPAS can learn operations from natural language, without the need to specify them in some formalism. This is implemented by extending BERT's architecture (Devlin et al., 2019) with additional embeddings that capture tabular structure, and with two classification layers for selecting cells and predicting a corresponding aggregation operator.
TaPas: Weakly Supervised Table Parsing via Pre-training
Importantly, we introduce a pre-training method for TAPAS, crucial for its success on the end task. We extend BERT's masked language model objective to structured data, and pre-train the model over millions of tables and related text segments crawled from Wikipedia. During pre-training, the model masks some tokens from the text segment and from the table itself, where the objective is to predict the original masked token based on the textual and tabular context.
TaPas: Weakly Supervised Table Parsing via Pre-training
Finally, we present an end-to-end differentiable training recipe that allows TAPAS to train from weak supervision. For examples that only involve selecting a subset of the table cells, we directly train the model to select the gold subset. For examples that involve aggregation, the relevant cells and the aggregation operation are not known from the denotation. In this case, we calculate an expected soft scalar outcome over all aggregation operators given the current model, and train the model with a regression loss against the gold denotation.
TaPas: Weakly Supervised Table Parsing via Pre-training
In comparison to prior attempts to reason over tables without generating logical forms (Neelakantan et al., 2015;Yin et al., 2016;Müller et al., 2019), TAPAS achieves better accuracy, and holds several advantages: its architecture is simpler as it includes a single encoder with no auto-regressive decoding, it enjoys pre-training, tackles more question types such as those that involve aggregation, and directly handles a conversational setting.
TaPas: Weakly Supervised Table Parsing via Pre-training
We find that on three different semantic parsing datasets, TAPAS performs better or on par in comparison to other semantic parsing and question answering models. On the conversational SQA (Iyyer et al., 2017), TAPAS improves stateof-the-art accuracy from 55.1 to 67.2, and achieves on par performance on WIKISQL (Zhong et al., 2017) and WIKITQ (Pasupat and Liang, 2015). Transfer learning, which is simple in TAPAS, from WIKISQL to WIKITQ achieves 48.7 accuracy, 4.2 points higher than state-of-the-art. Our code and pre-trained model are publicly available at https: //github.com/google-research/tapas.
TaPas: Weakly Supervised Table Parsing via Pre-training
Our model's architecture ( Figure 1) is based on BERT's encoder with additional positional embeddings used to encode tabular structure (visualized in Figure 2). We flatten the table into a sequence of words, split words into word pieces (tokens) and concatenate the question tokens before the table tokens. We additionally add two classification layers for selecting table cells and aggregation operators that operate on the cells. We now describe these modifications and how inference is performed. Additional embeddings We add a separator token between the question and the table, but unlike Hwang et al. (2019) not between cells or rows. Instead, the token embeddings are combined with table-aware positional embeddings before feeding them to the model. We use different kinds of positional embeddings:
TaPas: Weakly Supervised Table Parsing via Pre-training
• Position ID is the index of the token in the flattened sequence (same as in BERT).
TaPas: Weakly Supervised Table Parsing via Pre-training
• Segment ID takes two possible values: 0 for the question, and 1 for the table header and cells.
TaPas: Weakly Supervised Table Parsing via Pre-training
• Column / Row ID is the index of the column/row that this token appears in, or 0 if the token is a part of the question.
TaPas: Weakly Supervised Table Parsing via Pre-training
• Rank ID if column values can be parsed as floats or dates, we sort them accordingly and assign an embedding based on their numeric rank (0 for not comparable, 1 for the smallest item, i + 1 for an item with rank i). This can assist the model when processing questions that involve superlatives, as word pieces may not represent numbers informatively (Wallace et al., 2019).
TaPas: Weakly Supervised Table Parsing via Pre-training
• Previous Answer given a conversational setup where the current question might refer to the previous question or its answers (e.g., question 5 in Figure 3), we add a special embedding that marks whether a cell token was the answer to the previous question (1 if the token's cell was an answer, or 0 otherwise).  Figure 2: Encoding of the question "query?" and a simple table using the special embeddings of TAPAS. The previous answer embeddings are omitted for brevity.
TaPas: Weakly Supervised Table Parsing via Pre-training
aggregation operator, these cells can be the final answer or the input used to compute the final answer. Cells are modelled as independent Bernoulli variables. First, we compute the logit for a token using a linear layer on top of its last hidden vector. Cell logits are then computed as the average over logits of tokens in that cell. The output of the layer is the probability p (c) s to select cell c. We additionally found it useful to add an inductive bias to select cells within a single column. We achieve this by introducing a categorical variable to select the correct column. The model computes the logit for a given column by applying a new linear layer to the average embedding for cells appearing in that column. We add an additional column logit that corresponds to selecting no column or cells.
TaPas: Weakly Supervised Table Parsing via Pre-training
We treat this as an extra column with no cells. The output of the layer is the probability p (co) col to select column co computed using softmax over the column logits. We set cell probabilities p (c) s outside the selected column to 0.
TaPas: Weakly Supervised Table Parsing via Pre-training
Aggregation operator prediction Semantic parsing tasks require discrete reasoning over the table, such as summing numbers or counting cells. To handle these cases without producing logical forms, TAPAS outputs a subset of the table cells together with an optional aggregation operator. The aggregation operator describes an operation to be applied to the selected cells, such as SUM, COUNT, AVERAGE or NONE. The operator is selected by a linear layer followed by a softmax on top of the final hidden vector of the first token (the special [CLS] token). We denote this layer as p a (op), where op is some aggregation operator.
TaPas: Weakly Supervised Table Parsing via Pre-training
Inference We predict the most likely aggregation operator together with a subset of the cells (using the cell selection layer). To predict a discrete cell selection we select all table cells for which their probability is larger than 0.5. These predictions are then executed against the table to retrieve the answer, by applying the predicted aggregation over the selected cells.
TaPas: Weakly Supervised Table Parsing via Pre-training
Following the recent success of pre-training models on textual data for natural language understanding tasks, we wish to extend this procedure to structured data, as an initialization for our table parsing task. To this end, we pre-train TAPAS on a large number of tables from Wikipedia. This allows the model to learn many interesting correlations between text and the table, and between the cells of a columns and their header.
TaPas: Weakly Supervised Table Parsing via Pre-training
We create pre-training inputs by extracting texttable pairs from Wikipedia. We extract 6.2M tables: 3.3M of class Infobox 1 and 2.9M of class WikiTable. We consider tables with at most 500 cells. All of the end task datasets we experiment with only contain horizontal tables with a header row with column names. Therefore, we only extract Wiki tables of this form using the <th> tag to identify headers. We furthermore, transpose Infoboxes into a table with a single header and a single data row. The tables, created from Infoboxes, are arguably not very typical, but we found them to improve performance on the end tasks.
TaPas: Weakly Supervised Table Parsing via Pre-training
As a proxy for questions that appear in the end tasks, we extract the table caption, article title, article description, segment title and text of the segment the table occurs in as relevant text snippets. In this way we extract 21.3M snippets.
TaPas: Weakly Supervised Table Parsing via Pre-training
We convert the extracted text-table pairs to pretraining examples as follows: Following Devlin et al. (2019), we use a masked language model pre-training objective. We also experimented with adding a second objective of predicting whether the table belongs to the text or is a random table but did not find this to improve the performance on the end tasks. This is aligned with Liu et al. (2019) that similarly did not benefit from a next sentence prediction task.
TaPas: Weakly Supervised Table Parsing via Pre-training
For pre-training to be efficient, we restrict our word piece sequence length to a certain budget (e.g., we use 128 in our final experiments). That is, the combined length of tokenized text and table cells has to fit into this budget. To achieve this, we randomly select a snippet of 8 to 16 word pieces from the associated text. To fit the table, we start by only adding the first word of each column name and cell. We then keep adding words turn-wise until we reach the word piece budget. For every table we generate 10 different snippets in this way.
TaPas: Weakly Supervised Table Parsing via Pre-training
We follow the masking procedure introduced by BERT. We use whole word masking 2 for the text, and we find it beneficial to apply whole cell masking (masking all the word pieces of the cell if any of its pieces is masked) to the table as well.
TaPas: Weakly Supervised Table Parsing via Pre-training
We note that we additionally experimented with data augmentation, which shares a similar goal to pre-training. We generated synthetic pairs of questions and denotations over real tables via a grammar, and augmented these to the end tasks training data. As this did not improve end task performance significantly, we omit these results.
TaPas: Weakly Supervised Table Parsing via Pre-training
Overview We formally define table parsing in a weakly supervised setup as follows. Given a train-
TaPas: Weakly Supervised Table Parsing via Pre-training
, where x i is an utterance, T i is a table and y i is a corresponding set of denotations, our goal is to learn a model that maps a new utterance x to a program z, such that when z is executed against the corresponding table T , it yields the correct denotation y. The program z comprises a subset of the table cells and an optional aggregation operator. The table T maps a table cell to its value.
TaPas: Weakly Supervised Table Parsing via Pre-training
As a pre-processing step described in Section 5.1, we translate the set of denotations y for each example to a tuple (C, s) of cell coordinates C and a scalar s, which is only populated when y is a single scalar. We then guide training according to the content of (C, s). where s is populated but C is empty, we train the model to predict an aggregation over the table cells that amounts to s. We now describe each of these cases in detail.
TaPas: Weakly Supervised Table Parsing via Pre-training
Cell selection In this case y is mapped to a subset of the table cell coordinates C (e.g., question 1 in Figure 3). For this type of examples, we use a hierarchical model that first selects a single column and then cells from within that column only.
TaPas: Weakly Supervised Table Parsing via Pre-training
We directly train the model to select the column col which has the highest number of cells in C. For our datasets cells C are contained in a single column and so this restriction on the model provides a useful inductive bias. If C is empty we select the additional empty column corresponding to empty cell selection. The model is then trained to select cells C ∩ col and not select (T \ C) ∩ col. The loss is composed of three components: (1) the average binary cross-entropy loss over column selections:
TaPas: Weakly Supervised Table Parsing via Pre-training
where the set of columns Cols includes the additional empty column, CE(·) is the cross entropy loss, 1 is the indicator function.
TaPas: Weakly Supervised Table Parsing via Pre-training
(2) the average binary cross-entropy loss over column cell selections:
TaPas: Weakly Supervised Table Parsing via Pre-training
where Cells(col) is the set of cells in the chosen column. (3) As for cell selection examples no aggregation occurs, we define the aggregation supervision to be NONE (assigned to op 0 ), and the aggregation loss is:
TaPas: Weakly Supervised Table Parsing via Pre-training
The total loss is then J CS = J columns + J cells + αJ aggr , where α is a scaling hyperparameter.
TaPas: Weakly Supervised Table Parsing via Pre-training
Scalar answer In this case y is a single scalar s which does not appear in the table (i.e. C = ∅, e.g., question 2 in Figure 3). This usually corresponds to examples that involve an aggregation over one or more  the scalar answer s. To train the model given this form of supervision one could search offline (Dua et al., 2019;Andor et al., 2019) or online (Berant et al., 2013;Liang et al., 2018) for programs (table cells and aggregation) that execute to s. In our table parsing setting, the number of spurious programs that execute to the gold scalar answer can grow quickly with the number of table cells (e.g., when s = 5, each COUNT over any five cells is potentially correct). As with this approach learning can easily fail, we avoid it. Instead, we make use of a training recipe where no search for correct programs is needed. Our approach results in an end-to-end differentiable training, similar in spirit to Neelakantan et al. (2015). We implement a fully differentiable layer that latently learns the weights for the aggregation prediction layer p a (·), without explicit supervision for the aggregation type.
TaPas: Weakly Supervised Table Parsing via Pre-training
Specifically, we recognize that the result of executing each of the supported aggregation operators is a scalar. We then implement a soft differentiable estimation for each operator (Table 1), given the  token selection probabilities and the table values: compute(op, p s , T ). Given the results for all aggregation operators we then calculate the expected result according to the current model:
TaPas: Weakly Supervised Table Parsing via Pre-training
is a probability distribution normalized over aggregation operators excluding NONE.
TaPas: Weakly Supervised Table Parsing via Pre-training
We then calculate the scalar answer loss with Huber loss (Huber, 1964) given by:
TaPas: Weakly Supervised Table Parsing via Pre-training
compute(SUM,p s ,T ) compute(COUNT,p s ,T ) where a = |s pred − s|, and δ is a hyperparameter. Like Neelakantan et al. (2015), we find this loss is more stable than the squared loss. In addition, since a scalar answer implies some aggregation operation, we also define an aggregation loss that penalizes the model for assigning probability mass to the NONE class:
TaPas: Weakly Supervised Table Parsing via Pre-training
The total loss is then J SA = J aggr +βJ scalar , where β is a scaling hyperparameter. As for some examples J scalar can be very large, which leads to unstable model updates, we introduce a cutoff hyperparameter. Then, for a training example where J scalar > cutoff, we set J = 0 to ignore the example entirely, as we noticed this behaviour correlates with outliers. In addition, as computation done during training is continuous, while that being done during inference is discrete, we further add a temperature that scales token logits such that p s would output values closer to binary ones.
TaPas: Weakly Supervised Table Parsing via Pre-training
Ambiguous answer A scalar answer s that also appears in the table (thus C = ∅) is ambiguous, as in some cases the question implies aggregation (question 3 in Figure 3), while in other cases a  cell should be predicted (question 4 in Figure 3). Thus, in this case we dynamically let the model choose the supervision (cell selection or scalar answer) according to its current policy. Concretely, we set the supervision to be of cell selection if p a (op 0 ) ≥ S, where 0 < S < 1 is a threshold hyperparameter, and the scalar answer supervision otherwise. This follows hard EM (Min et al., 2019), as for spurious programs we pick the most probable one according to the current model.
TaPas: Weakly Supervised Table Parsing via Pre-training
We experiment with the following semantic parsing datasets that reason over single tables (see Table 2).
TaPas: Weakly Supervised Table Parsing via Pre-training
WIKITQ (Pasupat and Liang, 2015) This dataset consists of complex questions on Wikipedia tables. Crowd workers were asked, given a table, to compose a series of complex questions that include comparisons, superlatives, aggregation or arithmetic operation. The questions were then verified by other crowd workers.
TaPas: Weakly Supervised Table Parsing via Pre-training
SQA (Iyyer et al., 2017) This dataset was constructed by asking crowd workers to decompose a subset of highly compositional questions from WIKITQ, where each resulting decomposed question can be answered by one or more table cells.
TaPas: Weakly Supervised Table Parsing via Pre-training
The final set consists of 6, 066 question sequences (2.9 question per sequence on average).
TaPas: Weakly Supervised Table Parsing via Pre-training
WIKISQL (Zhong et al., 2017) This dataset focuses on translating text to SQL. It was constructed by asking crowd workers to paraphrase a templatebased question in natural language. Two other crowd workers were asked to verify the quality of the proposed paraphrases.
TaPas: Weakly Supervised Table Parsing via Pre-training
As our model predicts cell selection or scalar answers, we convert the denotations for each dataset to question, cell coordinates, scalar answer triples. SQA already provides this information (gold cells for each question). For WIKISQL and WIKITQ, we only use the denotations. Therefore, we derive cell coordinates by matching the denotations against the table contents. We fill scalar answer information if the denotation contains a single element that can be interpreted as a float, otherwise we set its value to NaN. We drop examples if there is no scalar answer and the denotation can not be found in the table, or if some denotation matches multiple cells.
TaPas: Weakly Supervised Table Parsing via Pre-training
We apply the standard BERT tokenizer on questions, table cells and headers, using the same vocabulary of 32k word pieces. Numbers and dates are parsed in a similar way as in the Neural Programmer (Neelakantan et al., 2017).
TaPas: Weakly Supervised Table Parsing via Pre-training
The official evaluation script of WIKITQ and SQA is used to report the denotation accuracy for these datasets. For WIKISQL, we generate the reference answer, aggregation operator and cell coordinates from the reference SQL provided using our own SQL implementation running on the JSON tables. However, we find that the answer produced by the official WIKISQL evaluation script is incorrect for approx. 2% of the examples. Throughout this paper we report accuracies against our reference answers, but we explain the differences and also provide accuracies compared to the official reference answers in Appendix A.
TaPas: Weakly Supervised Table Parsing via Pre-training
We start pre-training from BERT-Large (see Appendix B for hyper-parameters). We find it beneficial to start the pre-training from a pre-trained standard text BERT model (while randomly initializing our additional embeddings), as this enhances convergence on the held-out set.
TaPas: Weakly Supervised Table Parsing via Pre-training
We run both pre-training and fine-tuning on a setup of 32 Cloud TPU v3 cores with maximum sequence length 512. In this setup pre-training takes around 3 days and fine-tuning around 10 hours for WIKISQL and WIKITQ and 20 hours for SQA (with the batch sizes from table 12). The resource requirements of our model are essentially the same as BERT-large 3 .
TaPas: Weakly Supervised Table Parsing via Pre-training
For fine-tuning, we choose hyper-parameters using a black box Bayesian optimizer similar to Google Vizier (Golovin et al., 2017) for WIKISQL and WIKITQ. For SQA we use grid-search. We discuss the details in Appendix B.
TaPas: Weakly Supervised Table Parsing via Pre-training
Dev Test  
TaPas: Weakly Supervised Table Parsing via Pre-training
Pasupat and Liang (2015) 
TaPas: Weakly Supervised Table Parsing via Pre-training
All results report the denotation accuracy for models trained from weak supervision. We follow Niven and Kao (2019) and report the median for 5 independent runs, as BERT-based models can degenerate. We present our results for WIKISQL and WIKITQ in Tables 3 and 4 respectively. Table  3 shows that TAPAS, trained in the weakly supervised setting, achieves close to state-of-the-art performance for WIKISQL (83.6 vs 83.9 (Min et al., 2019)). If given the gold aggregation operators and selected cell as supervision (extracted from the reference SQL), which accounts as full supervision to TAPAS, the model achieves 86.4. Unlike the full SQL queries, this supervision can be annotated by non-experts. For WIKITQ the model trained only from the original training data reaches 42.6 which surpass similar approaches (Neelakantan et al., 2015). When we pre-train the model on WIKISQL or SQA (which is straight-forward in our setup, as we do not rely on a logical formalism), TAPAS achieves 48.7 and 48.8, respectively.   For SQA, Table 5 shows that TAPAS leads to substantial improvements on all metrics: Improving all metrics by at least 11 points, sequence accuracy from 28.1 to 40.4 and average question accuracy from 55.1 to 67.2. Table 6 shows an ablation study on our different embeddings. To this end we pretrain and fine-tune models with different features. As pre-training is expensive we limit it to 200, 000 steps. For all datasets we see that pre-training on tables and column and row embeddings are the most important. Positional and rank embeddings are also improving the quality but to a lesser extent.
TaPas: Weakly Supervised Table Parsing via Pre-training
We additionally find that when removing the scalar answer and aggregation losses (i.e., setting J SA=0 ) from TAPAS, accuracy drops for both datasets. For WIKITQ, we observe a substantial drop in performance from 29.0 to 23.1 when removing aggregation. For WIKISQL performance drops from 84.7 to 82.6. The relatively small decrease for WIKISQL can be explained by the fact that most examples do not need aggregation to be answered. In principle, 17% of the examples of the dev set have an aggregation (SUM, AVERAGE or COUNT), however, for all types we find that for more than 98% of the examples the aggregation is only applied to one or no cells. In the case of SUM and AVERAGE, this means that most examples can be answered by selecting one or no cells from the table. For COUNT the model without aggregation operators achieves 28.2 accuracy (by selecting 0 or 1 from the table) vs. 66.5 for the model with aggregation. Note that 0 and 1 are often found in a special index column. These properties of WIK-ISQL make it challenging for the model to decide whether to apply aggregation or not. For WIKITQ on the other hand, we observe a substantial drop in performance from 29.0 to 23.1 when removing aggregation.
TaPas: Weakly Supervised Table Parsing via Pre-training
Qualitative Analysis on WIKITQ We manually analyze 200 dev set predictions made by TAPAS on WIKITQ. For correct predictions via an aggregation, we inspect the selected cells to see if they match the ground truth. We find that 96% of the correct aggregation predictions where also correct in terms of the cells selected. We further find that 14% of the correct aggregation predictions had only one cell, and could potentially be achieved by cell selection, with no aggregation.
TaPas: Weakly Supervised Table Parsing via Pre-training
We also perform an error analysis and identify the following exclusive salient phenomena: (i) 12% are ambiguous ("Name at least two labels that released the group's albums."), have wrong labels or missing information ; (ii) 10% of the cases require complex temporal comparisons which could also not be parsed with a rich formalism such as SQL ("what country had the most cities founded in the 1830's?") ; (iii) in 16% of the cases the gold denotation has a textual value that does not appear in the table, thus it could not be predicted without performing string operations over cell values ; (iv) on 10%, the table is too big to fit in 512 tokens ; (v) on 13% of the cases TAPAS selected no cells, which suggests introducing penalties for this behaviour ; (vi) on 2% of the cases, the answer is the difference between scalars, so it is outside of the model capabilities ("how long did anne churchill/spencer live?") ; (vii) the other 37% of the cases could not be classified to a particular phenomenon.
TaPas: Weakly Supervised Table Parsing via Pre-training
Pre-training Analysis In order to understand what TAPAS learns during pre-training we analyze its performance on 10,000 held-out examples. We split the data such that the tables in the held-out  data do not occur in the training data. Table 7 shows the accuracy of masked word pieces of different types and in different locations. We find that average accuracy across position is relatively high (71.4). Predicting tokens in the header of the table is easiest (96.6), probably because many Wikipedia articles use instances of the same kind of table. Predicting word pieces in cells is a bit harder (63.4) than predicting pieces in the text (68.8). The biggest differences can be observed when comparing predicting words (74.1) and numbers (53.9). This is expected since numbers are very specific and often hard to generalize. The soft-accuracy metric and example (Appendix C) demonstrate, however, that the model is relatively good at predicting numbers that are at least close to the target.
TaPas: Weakly Supervised Table Parsing via Pre-training
Limitations TAPAS handles single tables as context, which are able to fit in memory. Thus, our model would fail to capture very large tables, or databases that contain multiple tables. In this case, the table(s) could be compressed or filtered, such that only relevant content would be encoded, which we leave for future work. In addition, although TAPAS can parse compositional structures (e.g., question 2 in Figure 3), its expressivity is limited to a form of an aggregation over a subset of table cells. Thus, structures with multiple aggregations such as "number of actors with an average rating higher than 4" could not be handled correctly. Despite this limitation, TAPAS succeeds in parsing three different datasets, and we did not encounter this kind of errors in Section 5.3. This suggests that the majority of examples in semantic parsing datasets are limited in their compositionality.
TaPas: Weakly Supervised Table Parsing via Pre-training
In this paper we presented TAPAS, a model for question answering over tables that avoids generating logical forms. We showed that TAPAS effectively pre-trains over large scale data of text-table pairs and successfully restores masked words and table cells. We additionally showed that the model can fine-tune on semantic parsing datasets, only using weak supervision, with an end-to-end differentiable recipe. Results show that TAPAS achieves better or competitive results in comparison to stateof-the-art semantic parsers.
TaPas: Weakly Supervised Table Parsing via Pre-training
In future work we aim to extend the model to represent a database with multiple tables as context, and to effectively handle large tables.
TaPas: Weakly Supervised Table Parsing via Pre-training
In some tables, WIKISQL contains "REAL" numbers stored in "TEXT" format. This leads to incorrect results for some of the comparison and aggregation examples. These errors in the WIK-ISQL execution accuracy penalize systems that do their own execution (rather then producing an SQL query). Table 8 shows two examples where our result derivation and the one used by WIK-ISQL differ because the numbers in the "Crowd" (col5) column are not represented as numbers in the respective SQL table. Table 9 and 10 contain accuracies compared against the official and our answers.
TaPas: Weakly Supervised Table Parsing via Pre-training
TAPAS ( 
TaPas: Weakly Supervised Table Parsing via Pre-training
In order to better understand how well the model predicts numbers, we relax our accuracy measure to a soft form of accuracy:
TaPas: Weakly Supervised Table Parsing via Pre-training
if x or y is not a number 1.0 − |x−y| max(x,y) else With this soft metric we get an overall accuracy of 74.5 (instead of 71.4) and an accuracy of 80.5 (instead of 53.9) for numbers. Showing that the model is pretty good at guessing numbers that are at least close to the target. The following example demonstrates this:  In the example, the model correctly restores the Draw (D) and Loss (L) numbers for Spain. It fails to restore the Points For (PF) and Points Against (PA) for Zimbabwe, but gives close estimates. Note that the model also does not produce completely consistent results for each row we should have PA + PD = PF and the column sums of PF and PA should equal.
TaPas: Weakly Supervised Table Parsing via Pre-training
Our approach to estimate aggregates of cells in the table operates directly on latent conditionally independent Bernoulli variables G c ∼ Bern(p c ) that indicate whether each cell is included in the aggregation and a latent categorical variable that indicates the chosen aggregation operation op: AVERAGE,   Table "2-10767641-15" from WIKISQL. "col6" was removed. The "Crowd" column is of type "REAL" but the cell values are actually stored as "TEXT". Below we have two questions from the training set with the answer that is produced by the WIKISQL evaluation script and the answer we derive.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Recently, semi-structured data (e.g. variable length tables without a fixed data schema) has attracted more attention because of its ubiquitous presence on the web. On a wide range of various table reasoning tasks, Transformer based architecture along with pretraining has shown to perform well (Eisenschlos et al., 2021;.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
In a nutshell, prior work used the Transformer architecture in a BERT like fashion by serializing tables or rows into word sequences (Yu et al., 2020    , where original position ids are used as positional information. Due to the usage of row/column ids and global position ids, prior strategies to linearize table structures introduced spurious row and column order biases (Herzig et al., 2020;Eisenschlos et al., , 2021Yin et al., 2020). Therefore, those models are vulnerable to row or column order perturbations. But, ideally, the model should make consistent predictions regardless of the row or column ordering for all practical purposes. For instance, in Figure 1, the predicted answer of TAPAS model (Herzig et al., 2020) for Question (a) "Of all song lengths, which one is the longest?" based on the original table is "5:00", which is incorrect. However, if the first row is adjusted to the end of the table during inference, the model gives the correct length "5:02" as an-swer. This probing example shows that the model being aware of row order information is inclined to select length values to the end of the table due to spurious training data bias. In our experiments on the SQA dataset, TAPAS models exhibit a 4% -6% (Section 5.2) absolute performance drop when facing such answer-invariant perturbations. Besides, most prior work (Chen et al., 2020;Yin et al., 2020) did not incorporate enough structural biases to models to address the limitation of sequential Transformer architecture, while others inductive biases which are either too strict Eisenschlos et al., 2021) or computationally expensive (Yin et al., 2020).
TableFormer: Robust Transformer Modeling for Table-Text Encoding
To this end, we propose TABLEFORMER, a Transformer architecture that is robust to row and column order perturbations, by incorporating structural biases more naturally. TABLEFORMER relies on 13 types of task-independent table↔text attention biases that respect the table structure and table-text relations. For Question (a) in Figure 1, TABLEFORMER could predict the correct answer regardless of perturbation, because the model could identify the same row information with our "same row" bias, avoiding spurious biases introduced by row and global positional embeddings. For Question (b), TAPAS predicted only partially correct answer, while TABLEFORMER could correctly predict "Spain, Ukraine" as answers. That's because our "cell to sentence" bias could help table cells ground to the paired sentence. Detailed attention bias types are discussed in Section 5.2.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Experiments on 3 table reasoning datasets show that TABLEFORMER consistently outperforms original TAPAS in all pretraining and intermediate pretraining settings with fewer parameters. Also, TABLEFORMER's invariance to row and column perturbations, leads to even larger improvement over those strong baselines when tested on perturbations. Our contributions are as follows:
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• We identified the limitation of current tabletext encoding models when facing row or column perturbation.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• We propose TABLEFORMER, which is guaranteed to be invariant to row and column order perturbations, unlike current models.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• TABLEFORMER encodes table-text structures better, leading to SoTA performance on SQA dataset, and ablation studies show the effectiveness of the introduced inductive biases.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
In this section, we discuss TAPAS which serves as the backbone of the recent state-of-the-art tabletext encoding architectures. TAPAS (Herzig et al., 2020) uses Transformer architecture in a BERT like fashion to pretrain and finetune on tabular data for Concretely, for any table-text linearized sequence S = {v 1 , v 2 , · · · , v n }, where n is the length of table-text sequence, the input to TAPAS is summation of embedding of the following: token ids (W ) = {w v 1 , w v 2 , · · · , w vn } positional ids (B) = {b 1 , b 2 , · · · , b n } segment ids (G) = {g seg 1 , g seg 2 , · · · , g segn } column ids (C) = {c col 1 , c col 2 , · · · , c coln } row ids (R) = {r row 1 , r row 2 , · · · , r rown } rank ids (Z) = {z rank 1 , z rank 2 , · · · , z rankn } where seg i , col i , row i , rank i correspond to the segment, column, row, and rank id for the ith token, respectively. As for the model, TAPAS uses BERT's selfattention architecture (Vaswani et al., 2017) offthe-shelf. Each Transformer layer includes a multihead self-attention sub-layer, where each token attends to all the tokens. Let the layer input H = [h 1 , h 2 , · · · , h n ] ∈ R n×d corresponding to S, where d is the hidden dimension, and h i ∈ R d×1 is the hidden representation at position i. For a single-head self-attention sub-layer, the input H is projected by three matrices W Q ∈ R d×d K , W K ∈ R d×d K , and W V ∈ R d×d V to the corresponding representations Q, K, and V :
TableFormer: Robust Transformer Modeling for Table-Text Encoding
529 Transformer (Self Attention) Transformer (Self Attention)
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Learnable structure enforced attention bias scalars. We use 13 types of attention biases based on task independent relation between header, row, column, text, etc.    Figure 1 and its paired question "query". Different colors in the attention bias matrix denote different types of task independent biases derived based on the table structure and the associated text.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Then, the output of this single-head selfattention sub-layer is calculated as:
TableFormer: Robust Transformer Modeling for Table-Text Encoding
3 TABLEFORMER: Robust Structural Table Encoding As shown in Figure 2, TABLEFORMER encodes the general table structure along with the associated text by introducing task-independent relative attention biases for Input of TABLEFORMER. TABLEFORMER uses the same token embeddings W , segment embeddings G, and rank embeddings Z as TAPAS. However, we make 2 major modifications: 1) No row or column ids. We do not use row embeddings R or column embeddings C to avoid any potential spurious row and column order biases.  2018)). TABLEFORMER does not consume any sort of column and row order information in the input. The main intuition is that, for cells in the table, the only useful positional information is whether two cells are in the same row or column and the column header of each cell, instead of the absolute order of the row and column containing them. Thus, inspired by relative positional encoding (Shaw et al., 2018) and graph encoding (Ying et al., 2021), we capture this with a same column/row relation as one kind of relative position between two linearized tokens. Similarly, we uses 12 such table-text structure relevant relations (including same cell, cell to header and so on) and one extra type representing all other relations not explicitly defined. All of them are introduced in the form of learnable attention bias scalars.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Formally, we consider a function φ(v i , v j ) : V × V → N, which measures the relation between v i and v j in the sequence (v i , v j ∈ S). The function φ can be defined by any relations between the tokens in the table-text pair.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Attention Biases in TABLEFORMER. In our work, φ(v i , v j ) is chosen from 13 bias types, corresponding to 13 table-text structural biases. The attention biases are applicable to any table-text pair and can be used for any downstream task:
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• "same row" identifies the same row information without ordered row id embedding or global positional embedding, which help the model to be invariant to row perturbations,
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• "same column", "header to column cell", and "cell to column header" incorporates the same column information without ordered column id embedding,
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• "cell to column header" makes each cell aware of its column header without repeated column header as features,
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• "header to sentence" and "cell to sentence" help column grounding and cell grounding of the paired text,
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• "sentence to header", "sentence to cell", and "sentence to sentence" helps to understand the sentence with the table as context,
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• "header to same header" and "header to other header" for better understanding of table schema, and "same cell bias" for cell content understanding.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Note that, each cell can still attend to other cells in the different columns or rows through "others" instead of masking them out strictly. We assign each bias type a learnable scalar, which will serve as a bias term in the self-attention module. Specifically, each self-attention head in each layer have a set of learnable scalars {b 1 , b 2 , · · · , b 13 } corresponding to all types of introduced biases. For one head in one self-attention sub-layer of TABLEFORMER, Equation 2 in the Transformer is replaced by:
TableFormer: Robust Transformer Modeling for Table-Text Encoding
whereĀ is a matrix capturing the similarity between queries and keys,Â is the Attention Bias Matrix,
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Relation between TABLEFORMER and ETC. ETC (Ainslie et al., 2020) uses vectors to represent relative position labels, although not directly applied to table-text pairs due to its large computational overhead (Eisenschlos et al., 2021). TABLE-FORMER differs from ETC in the following aspects (1) ETC uses relative positional embeddings while TABLEFORMER uses attention bias scalars. In practice, we observed that using relative positional embeddings increases training time by more than 7x, (2) ETC uses global memory and local attention, while TABLEFORMER uses pairwise attention without any global memory overhead, (3) ETC uses local sparse attention with masking, limiting its ability to attend to all tokens, (4) ETC did not explore  (3) and (4).
TableFormer: Robust Transformer Modeling for Table-Text Encoding
We use the following datasets in our experiments.  (Pasupat and Liang, 2015) and Sequential QA (SQA) (Iyyer et al., 2017) datasets. WTQ was crowd-sourced based on complex questions on Wikipedia tables. SQA is composed of 6, 066 question sequences (2.9 question per sequence on average), constructed by decomposing a subset of highly compositional WTQ questions.  (Chen et al., 2020), where the tables were extracted from Wikipedia and the sentences were written by crowd workers. Among total 118, 000 sentences, each one is a positive (entailed) or negative sentence.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Perturbation Evaluation Set. For SQA and TABFACT, we also created new test sets to measure models' robustness to answer-invariant row and column perturbations during inference. Specifically, row and column orders are randomly perturbed for all tables in the standard test sets. 2
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Pre-training All the models are first tuned on the Wikipidia text-table pretraining dataset (Herzig et al., 2020), optionally tuned on synthetic dataset at an intermediate stage ("inter") , and finally fine-tuned on the target dataset. To get better performance on WTQ, we follow Herzig et al. (2020) to further pretrain on SQA dataset after the intermediate pretraining stage in the "inter-sqa" setting.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Evaluation For SQA, we report the cell selection accuracy for all questions (ALL) using the official evaluation script, cell selection accuracy for all sequences (SEQ), and the denotation accuracy for all questions (ALL d ). To evaluate the models' robustness in the instance level after perturbations, we also report a lower bound of example prediction variation percentage:
TableFormer: Robust Transformer Modeling for Table-Text Encoding
where t2t, t2f, f2t, and f2f represents how many example predictions turning from correct to correct, from correct to incorrect, from incorrect to correct and from incorrect to incorrect, respectively, after perturbation. We report denotation accuracy on WTQ and binary classification accuracy on TAB-FACT respectively.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
We use TAPAS BASE and TAPAS LARGE as baselines, where Transformer architectures are exactly same as BERT BASE and BERT LARGE (Devlin et al., 2019), and parameters are initialized from BERT BASE and BERT LARGE respectively. Correspondingly, we have our TABLEFORMER BASE and TABLEFORMER LARGE , where attention bias scalars are initialized to zero, and all other parameters are initialized from BERT BASE and BERT LARGE .
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Could we alleviate the spurious ordering biases by data augmentation alone, without making any modeling changes? To answer this, we train another set of models by augmenting the training data  
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Besides standard testing results to compare   • Can we use perturbation based data augmentation to achieve robustness at test time?
TableFormer: Robust Transformer Modeling for Table-Text Encoding
• Which attention biases in TABLEFORMER contribute the most to performance?  Table 2 shows that TABLEFORMER also outperforms TAPAS baseline models in all settings, which shows the effectiveness of TABLE-FORMER in the table entailment task. Note that,  is not comparable to our results, be-cause they used different pretraining data, different pretraining objectives, and BART NLG model instead of BERT NLU model. But TABLEFORMER attention bias is compatible with BART model.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
One of our major contributions is to systematically evaluate models' performance when facing row and column order perturbation in the testing stage.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Ideally, model predictions should be consistent on table QA and entailment tasks when facing such perturbation, because the table semantics remains the same after perturbation.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
However, in Table 1 and 2, we can see that in our perturbed test set, performance of all TAPAS models drops significantly in both tasks. TAPAS models drops by at least 3.7% and up to 6.5% in all settings on SQA dataset in terms of ALL accuracy, while our TABLEFORMER being strictly invariant to such row and column order perturbation leads to no drop in performance. 4 Thus, in the perturbation setting, TABLEFORMER outperforms all TAPAS baselines even more significantly, with at least 6.2% and 2.4% improvements on SQA and TABFACT dataset, respectively. In the instance level, we can see that, with TAPAS, there are many example predictions changed due to high V P , while there is nearly no example predictions changed with TABLEFORMER (around zero V P ).  
TableFormer: Robust Transformer Modeling for Table-Text Encoding
We compare the model sizes of TABLEFORMER and TAPAS in Table 4. We added only a few attention bias scalar parameters (13 parameters per head per layer) in TABLEFORMER, which is negligible compared with the BERT model size. Meanwhile, we delete two large embedding metrics (512 row ids and 512 column ids). Thus, TABLEFORMER outperforms TAPAS with fewer parameters.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
In this section, we experiment with several variants of TABLEFORMER to understand the effectiveness of its submodules. The performance of all variants of TAPAS and TABLEFORMER that we tried on the SQA development set is shown in Table 5.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Learnable Attention Biases v/s Masking. Instead of adding learnable bias scalars, we mask out some attention scores to restrict attention to those tokens in the same columns and rows, as well as the paired sentence, similar to  (SAT). We can see that TAPAS BASE-SAT performs worse than TAPAS BASE , which means that restricting attention to only same columns and rows by masking reduce the modeling capacity. This led to choosing soft bias addition over hard masking.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Attention Bias Scaling. Unlike TABLE-FORMER, we also tried to add attention biases before the scaling operation in the self-attention module (SO). Specifically, we compute pair-wise attention score by:
TableFormer: Robust Transformer Modeling for Table-Text Encoding
instead of using:  Table 5: ALL questions' cell selection accuracy of TABLEFORMER variants on SQA development set. rcgp represents the setting including row ids, column ids and global positional ids, c-gp represents column ids and global positional ids, gp represents global positional ids, and pcp represents per-cell positional ids. "SAT" represents masking out some attention scores. "SO" represents adding attention bias before scaling.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
which is the element-wise version of Equation 1 and 3. However, Table 5 shows that TABLEFORMER BASE-SO performs worse than TABLEFORMER BASE , showing the necessity of adding attention biases after the scaling operation. We think the reason is that the attention bias term does not require scaling, because attention bias scalar magnitude is independent of d K , while the dot products grow large in magnitude for large values of d K . Thus, such bias term could play an more important role without scaling, which helps each attention head know clearly what to pay more attention to according to stronger inductive biases.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
With TAPAS BASE , TABLEFORMER BASE-SO , and TABLEFORMER BASE , we first tried the full-version where row ids, column ids, and global positional ids exist as input (rc-gp). Then, we deleted row ids (c-gp), and column ids (gp) sequentially. Finally, we changed global positional ids in gp to per-cell positional ids (pcp). Table 5 shows that TAPAS BASE performs significantly worse from rcgp → c-gp → gp → pcp, because table structure information are deleted sequentially during such process. However, with TABLEFORMER BASE , there is no obvious performance drop during the same process. That shows the structural inductive biases in TABLEFORMER can provide complete table structure information. Thus, row ids, column ids and global positional ids are not necessary in  
TableFormer: Robust Transformer Modeling for Table-Text Encoding
As stated in Section 4.3, perturbing row and column orders as augmented data during training can serve as another possible solution to alleviate the spurious row/column ids bias. We can see that the performance of TAPAS BASE on SQA dataset improves with such augmentation. Also, as the number of perturbed versions of each table increases, model performance first increases and then decreases, reaching the best results with 8 perturbed versions. We suspect that too many versions of the same table confuse the model about different row and column ids for the same table, leading to decreased performance from 8p to 16p. Despite its usefulness, such data perturbation is still worse than TABLEFORMER, because it could not incorporate other relevant text-table structural inductive biases like TABLEFORMER.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Although, such data augmentation makes the model more robust to row and column order perturbation with smaller V P compared to standard TAPAS BASE , there is still a significant prediction drift after perturbation. As shown in Table 6, V P decreases from 1p to 16p, however, the best V P (7.0%) is still much higher than (nearly) no variation (0.1%) of TABLEFORMER.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
To sum up, TABLEFORMER is superior to row and column order perturbation augmentation, because of its additional structural biases and strictly consistent predictions after perturbation.  
TableFormer: Robust Transformer Modeling for Table-Text Encoding
We conduct ablation study to demonstrate the utility of all 12 types of defined attention biases. For each ablation, we set the corresponding attention bias type id to "others" bias id. Table 7 shows TAPAS BASE 's performance SQA dev set. Overall, all types of attention biases help the TABLE-FORMER performance to some extent, due to certain performance drop after deleting each bias type. Amongst all the attention biases, deleting "same row" bias leads to most significant performance drop, showing its crucial role for encoding table row structures. There is little performance drop after deleting "same column" bias, that's because TABLEFORMER could still infer the same column information through "cell to its column header" and "header to its column cell" biases. After deleting all same column information ("same column", "cell to column header" and "header to column cell" biases), TABLEFORMER performs significantly worse without encoding column structures. Similarly, there is little performance drop after deleting "same cell" bias, because TABLEFORMER can still infer same cell information through "same row" and "same column" biases.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
TABLEFORMER increases the training time by around 20%, which might not be ideal for very long tables and would require a scoped approach. Secondly, with the strict row and column order invariant property, TABLEFORMER cannot deal with questions based on absolute orders of rows in tables. This however is not a practical requirement based on the current dataset. Doing a manual study of 1800 questions in SQA dataset, we found that there are 4 questions 5 (0.2% percentage) whose answers depend on orders of rows. Three of them asked "which one is at the top of the table", another asks "which one is listed first". However, these questions could be potentially answered by adding back row and column order information based on TABLEFORMER.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Transformers for Tabular Data. Yin et al. (2020) prepended corresponding column headers to cells contents, and Chen et al. (2020) used corresponding column headers as features for cells. However, such methods encode each table header multiple times, leading to duplicated computing overhead. Also, tabular structures (e.g. same row information) are not fully incorporated to such models. Meanwhile, Yin et al. (2020) leveraged row encoder and column encoder sequentially, which introduced much computational overhead, thus requiring retrieving some rows as a preprocessing step. Finally, SAT , Deng et al. (2021) and Wang et al. (2021) restricted attention to same row or columns with attention mask, where such inductive bias is too strict that cells could not directly attend to those cells in different row and columns, hindering the modeling ability according to Table 5.  used the seq2seq BART generation model with a standard Transformer encoder-decoder architecture. In all models mentioned above, spurious inter-cell order biases still exist due to global positional ids of Transformer, leading to the vulnerability to row or column order perturbations, while our TABLE-FORMER could avoid such problem. Mueller et al. (2019) and  also used relative positional encoding to encode table structures, but they modeled the relations as learnable relation vectors, whose large overhead prevented pretraining and led to poor performance without pretraining, similarly to ETC (Ainslie et al., 2020) explained in Section 3.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
Structural and Relative Attention. Modified attention scores has been used to model relative positions (Shaw et al., 2018), long documents (Dai et al., 2019;Beltagy et al., 2020;Ainslie et al., 2020), and graphs (Ying et al., 2021). But adding 5 We find such 4 questions by manually looking at all 125 questions where the model predictions turn from correct to incorrect after replacing TAPASLARGE with TABLEFORMERLARGE.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
learnable attention biases to model tabular structures has been under-explored.
TableFormer: Robust Transformer Modeling for Table-Text Encoding
In this paper, we identified the vulnerability of prior table encoding models along two axes: (a) capturing the structural bias, and (b) robustness to row and column perturbations. To tackle this, we propose TABLEFORMER, where learnable task-independent learnable structural attention biases are introduced, while making it invariant to row/column order at the same time. Experimental results showed that TABLEFORMER outperforms strong baselines in 3 table reasoning tasks, achieving state-of-the-art performance on SQA dataset, especially when facing row and column order perturbations, because of its invariance to row and column orders.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Humans are voracious information omnivores, consuming information in a number of forms. Unstructured text is the form covered in most work in NLP, but another form widely used on the web, academic papers, and reports is the table, where elements are organized in rows and columns and presented in a structured and succinct way. Because of this, systems to aid information access over tables, such as table-based question answering (QA) (Pasupat and Liang, 2015;Iyyer et al., 2017;Zhong et al., 2017), hold significant promise. However, they also require understanding of the table structure and sophisticated reasoning across multiple elements to get the final answer. This intrinsic complexity not only often requires special-purpose model designs such as pipeline systems that generate structured queries as intermediate steps (Liang et al., 2018;Wang et al., 2019;Yin et al., 2020;Yu et al., 2021), but also adds an extra burden to the process of data annotation (Pasupat and Liang, 2015;Shi et al., 2020).
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Humans are voracious information omnivores, consuming information in a number of forms. Unstructured text is the form covered in most work in NLP, but another form widely used on the web, academic papers, and reports is the table, where elements are organized in rows and columns and presented in a structured and succinct way. Because of this, systems to aid information access over tables, such as table-based question answering (QA) (Pasupat and Liang, 2015;Iyyer et al., 2017;Zhong et al., 2017), hold significant promise. However, they also require understanding of the table structure and sophisticated reasoning across multiple elements to get the final answer. This intrinsic complexity not only often requires special-purpose model designs such as pipeline systems that generate structured queries as intermediate steps (Liang et al., 2018;Wang et al., 2019;Yin et al., 2020;Yu et al., 2021), but also adds an extra burden to the process of data annotation (Pasupat and Liang, 2015;Shi et al., 2020).
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Given the challenges above, we ask the question: "can we create a simple model that is able to answer complex table-based questions with minimal annotation effort?" Both modeling simplicity and limited assumptions regarding data availability would make it easier to apply table-based QA systems in practical scenarios. To this end, we focus on developing a simple and generic end2end table-based QA model where only a limited number of annotated natural language (NL) questions are available; the first attempt to address this problem under few-shot settings. In order to answer table-related questions, an end2end model needs to understand both the NL question and the table, build connections between the two formats, then perform complex reasoning. Taking the manually annotated question in Fig. 1 as an example, models need to align entities ("Collateral Damage") and concepts ("film", "air") in the question to elements in the table (the "Collateral Damage" cell and the "Film" and "Date" columns) and perform comparative reasoning based on chronological order (indicated by "previous" and "before") to find the final answer. Motivated by this, we propose OmniTab, an omnivorous pretraining approach that consumes natural data to endow models with the ability to understand and align NL with tables, and synthetic data to train models to perform reasoning.   Table] P r e Synthetic question [Table] Pre-train OmniTab Figure 2: The overall framework of generating and pretraining with natural data (blue pipeline), synthetic data (green pipeline), and finetuning with limited annotated questions (orange pipeline) for our OmniTab model.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
To obtain natural NL-table pairs, we propose a novel approach that leverages the multitude of tables freely available from the web, and uses retrieval to pair them with relevant NL sentences. Compared with manually defined heuristics used in previous work (Herzig et al., 2020;Yin et al., 2020), retrieval-based methods have the potential to discover better-aligned sentences. We explore different retrieval methods including string-based matching, sparse retrieval, and dense retrieval (Karpukhin et al., 2020;Khattab and Zaharia, 2020;. Given these retrieved pairs, phrases in the sentence that align with table elements are then masked and the model takes both the masked sentence and the linearized table as input to predict masked mentions. For example, in Fig. 1 the retrieved sentence describes a particular row and contains two phrases matching cells in the table (i.e., "Spider-Man" and "$114.8 million") which are masked for prediction. To perform this sort of masked prediction, models need to understand that the sentence is about a record-breaking movie and refer to the table to extract the correct cells. Thus, training on this data endows models with better understanding and alignment across both formats.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
To obtain natural NL-table pairs, we propose a novel approach that leverages the multitude of tables freely available from the web, and uses retrieval to pair them with relevant NL sentences. Compared with manually defined heuristics used in previous work (Herzig et al., 2020;Yin et al., 2020), retrieval-based methods have the potential to discover better-aligned sentences. We explore different retrieval methods including string-based matching, sparse retrieval, and dense retrieval (Karpukhin et al., 2020;Khattab and Zaharia, 2020;. Given these retrieved pairs, phrases in the sentence that align with table elements are then masked and the model takes both the masked sentence and the linearized table as input to predict masked mentions. For example, in Fig. 1 the retrieved sentence describes a particular row and contains two phrases matching cells in the table (i.e., "Spider-Man" and "$114.8 million") which are masked for prediction. To perform this sort of masked prediction, models need to understand that the sentence is about a record-breaking movie and refer to the table to extract the correct cells. Thus, training on this data endows models with better understanding and alignment across both formats.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
For the synthetic data approach, we propose a method where structured queries such as SQL are first sampled and then converted into NL questions using an SQL2NL model, which allows for control of the reasoning operations covered by the SQL. Compared to existing work that trains directly on SQL (Liu et al., 2021), an approach hindered by the gap between structured and natural language, training on synthetic NL questions can close the gap, especially when limited annotated NL questions are available. We train the SQL2NL model with a small number of SQL-NL pairs and further boost its performance using verification-based selftraining, which selects high-quality generated NL questions based on their likelihood to generate the gold answer. The converted NL question concatenated with the linearized table is fed into the model to directly generate the final answer, as shown in the synthetic example in Fig. 1 which involves comparative reasoning indicated by the phrase "more than". Although the fluency and naturalness of synthetic data is usually lower than natural data, learning on synthetic data provides a direct way to simulate different reasoning skills, which is relatively sparse in natural data.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Our overall framework is shown in Fig. 2. We use tables from Wikipedia and retrieve relevant sentences from the same page to generate natural text-table parallel data after masking mentions aligned to table elements (the blue pipeline on the left of Fig. 2). We use SQL queries sampled by Liu et al. (2021) and convert them to NL questions as synthetic text-table parallel data (the green pipeline on the right of Fig. 2). We use WikiTableQuestions (WTQ) (Pasupat and Liang, 2015), a widely used table-based QA dataset consisting of complex questions, as our major benchmark to evaluate our pretraining methods, and further use WikiSQL (Zhong et al., 2017) and topic-categorized WTQ (Chemmengath et al., 2021) to evaluate the robustness of our methods, all under few-shot setting with sizes ranging from 16 to 1,024. When using only 128 annotated questions, our model OmniTab improves over the best-performing baseline by an absolute gain of 13.2% and 12.3% with natural and synthetic pretraining separately and 16.2% when combined, demonstrating the effectiveness of the approach. We also achieve state-of-the-art performance on the full WTQ with an absolute gain of 2.7%. Extensive ablations and analyses reveal that natural and synthetic data indeed play the role of enhancing alignment and injecting reasoning, shedding light on future works on omnivorous pretraining.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Our overall framework is shown in Fig. 2. We use tables from Wikipedia and retrieve relevant sentences from the same page to generate natural text-table parallel data after masking mentions aligned to table elements (the blue pipeline on the left of Fig. 2). We use SQL queries sampled by Liu et al. (2021) and convert them to NL questions as synthetic text-table parallel data (the green pipeline on the right of Fig. 2). We use WikiTableQuestions (WTQ) (Pasupat and Liang, 2015), a widely used table-based QA dataset consisting of complex questions, as our major benchmark to evaluate our pretraining methods, and further use WikiSQL (Zhong et al., 2017) and topic-categorized WTQ (Chemmengath et al., 2021) to evaluate the robustness of our methods, all under few-shot setting with sizes ranging from 16 to 1,024. When using only 128 annotated questions, our model OmniTab improves over the best-performing baseline by an absolute gain of 13.2% and 12.3% with natural and synthetic pretraining separately and 16.2% when combined, demonstrating the effectiveness of the approach. We also achieve state-of-the-art performance on the full WTQ with an absolute gain of 2.7%. Extensive ablations and analyses reveal that natural and synthetic data indeed play the role of enhancing alignment and injecting reasoning, shedding light on future works on omnivorous pretraining.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
In this section, we first explain the setting of tablebased QA, then introduce the input format as well as our model architecture. 
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
and M columns, where the token sequence in the cell located at the i-th row and j-th column is denoted as c i,j . The first row r 1 is the header row, indicating the meaning of each column. Table-based QA aims to generate the answer a given both the question q and the table T as the input.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Input Format We follow Liu et al. (2021) in concatenating the NL context with the linearized table as input. We flatten the table following a topto-bottom and left-to-right order, where we prepend "col:" to the beginning of the header and "row i:" to the beginning of the i-th row to separate different rows: T = [col: r 1 row 1: r 2 ... row N : r N ]. Cells within each row are separated by a vertical bar "|"
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Finally, the question q is prepended to the linearized table: [q T ].
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Model Architecture We use the state-of-the-art table-based QA model TAPEX (Liu et al., 2021) as our base model, which is based on BART (Lewis et al., 2020b). It feeds questions and tables into the encoder and generates answers from the decoder: P (a|q, T ). Multiple answers are joined with commas into a single output sequence.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Model Architecture We use the state-of-the-art table-based QA model TAPEX (Liu et al., 2021) as our base model, which is based on BART (Lewis et al., 2020b). It feeds questions and tables into the encoder and generates answers from the decoder: P (a|q, T ). Multiple answers are joined with commas into a single output sequence.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
As mentioned in the introduction, Learning on real sentences can endow models to grasp the nuances in language and align with structured tables. On the other hand, synthetic data is flexible, manipulable, and easy to obtain. It is costless to generate synthetic data via manipulating different aspects of the generated data to incorporate various desired properties. As a result, we can generate large amounts of complicated synthetic data covering various reasoning operations, which is lacking in the NL corpora. This motivates us to explore both types of data.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Using the Wikipedia table corpus released by Herzig et al. (2020), we explore three retrieval methods to construct aligned NL- Table pairs and propose a new pretraining objective.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Using the Wikipedia table corpus released by Herzig et al. (2020), we explore three retrieval methods to construct aligned NL- Table pairs and propose a new pretraining objective.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Since sentences relevant to a table are usually included in the same document, we restrict our retrieval models to only consider the document containing the table, with the purpose of reducing noise and increasing efficiency.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
String-based Matching For each sentence, we enumerate over all cells in the table and find the longest common substring (LCS) between a cell c i,j and a sentence s. An LCS is considered as a mention to be masked if it (1) is not a stopword, (2) contains alphanumeric characters, (3) is a complete word, and (4) is longer than 70% of the length of the cell. We choose the sentence with the largest number of mentions to pair with the table.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Sparse Retrieval with BM25 Another method of string-based matching is BM25 with tables as queries and sentences as candidates. Different from the above method matching whole cells, BM25 treats tables as a bag of tokens. We linearize tables as queries and choose the most relevant sentence to compose the pair. Since BM25 retrieval does not return aligned phrases and cells, we resort to the above method detect mentions.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
The above two methods can only consider exact string matches, which is sub-optimal because different expressions might be used between sentences and tables such as "$114,844,116" and "$114.8 million" in Fig. 1. Tables tend to use full and formal expressions, while expressions in NL tend to be casual, often with abbreviations. To address this issue, we propose to use dense representations to perform soft matching. Many works use a single dense vector to represent the whole query/document for retrieval (Guu et al., 2020;Karpukhin et al., 2020). However, in our case, queries are tables usually consisting of many cells, 2 thus representing a whole table as a single vector might lead to information loss, which performs well below BM25 in our preliminary experiments. Additionally, retrieval based on a single vector only returns sentence-table pairs without revealing phrase-cell alignment, which is required for masking purposes. Thus, we propose to use token representation to directly match phrases and cells, similar to token-level dense retrieval (Khattab and Zaharia, 2020; in spirit. Specifically, we use BART to obtain token representations for each sentence s and table T separately. We then use a named entity recognition (NER) model to detect candidate phrases {p l } L l=1 in the sentence. Each phrase and cell are represented as the average token representation, denoted as e p l and e c i,j respectively after normalized to a unit vector. We compute a similarity for each cellphrase pair using dot product e c i,j ·e p l , resulting in a two-dimensional similarity matrix A ∈ R N M ×L , where each row and column correspond to a cell and a phrase respectively. We aggregate the relevance matrix A to derive relevance scores for ranking sentences and an assessment score for each phrase to choose salient mentions for masking. Given the fact that soft match based on dense vectors usually yields a non-zero relevance score even for irrelevant pairs, we apply the max-sparsify op-2 Wiki tables have 26 cells on avg with extreme cases 500+. erator to emphasize relevant matches and eliminate noisy irrelevant matches, similarly to the max operator in Khattab and Zaharia (2020); . The max-sp(A, dim = i) keeps the max entry along dimension i of the matrix A and changes other entries to zero. As illustrated in Fig. 3, we first apply this operator over all phrases (dim = 1), assigning each cell the best-matching phrase, then apply it over all cells (dim = 0), assigning each remaining phrase to its best-matching cell. We use the sum of the sparsified matrix as the relevance score to choose the best-matching sentences, rank remaining phrases in that sentence (p 2 > p 3 > p 1 in Fig. 3), and choose phrases with scores higher than a threshold τ as mentions to be masked.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
The above two methods can only consider exact string matches, which is sub-optimal because different expressions might be used between sentences and tables such as "$114,844,116" and "$114.8 million" in Fig. 1. Tables tend to use full and formal expressions, while expressions in NL tend to be casual, often with abbreviations. To address this issue, we propose to use dense representations to perform soft matching. Many works use a single dense vector to represent the whole query/document for retrieval (Guu et al., 2020;Karpukhin et al., 2020). However, in our case, queries are tables usually consisting of many cells, 2 thus representing a whole table as a single vector might lead to information loss, which performs well below BM25 in our preliminary experiments. Additionally, retrieval based on a single vector only returns sentence-table pairs without revealing phrase-cell alignment, which is required for masking purposes. Thus, we propose to use token representation to directly match phrases and cells, similar to token-level dense retrieval (Khattab and Zaharia, 2020; in spirit. Specifically, we use BART to obtain token representations for each sentence s and table T separately. We then use a named entity recognition (NER) model to detect candidate phrases {p l } L l=1 in the sentence. Each phrase and cell are represented as the average token representation, denoted as e p l and e c i,j respectively after normalized to a unit vector. We compute a similarity for each cellphrase pair using dot product e c i,j ·e p l , resulting in a two-dimensional similarity matrix A ∈ R N M ×L , where each row and column correspond to a cell and a phrase respectively. We aggregate the relevance matrix A to derive relevance scores for ranking sentences and an assessment score for each phrase to choose salient mentions for masking. Given the fact that soft match based on dense vectors usually yields a non-zero relevance score even for irrelevant pairs, we apply the max-sparsify op-2 Wiki tables have 26 cells on avg with extreme cases 500+. erator to emphasize relevant matches and eliminate noisy irrelevant matches, similarly to the max operator in Khattab and Zaharia (2020); . The max-sp(A, dim = i) keeps the max entry along dimension i of the matrix A and changes other entries to zero. As illustrated in Fig. 3, we first apply this operator over all phrases (dim = 1), assigning each cell the best-matching phrase, then apply it over all cells (dim = 0), assigning each remaining phrase to its best-matching cell. We use the sum of the sparsified matrix as the relevance score to choose the best-matching sentences, rank remaining phrases in that sentence (p 2 > p 3 > p 1 in Fig. 3), and choose phrases with scores higher than a threshold τ as mentions to be masked.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Given a retrieved sentence s associated with the table T , we apply two masking strategies: (1) randomly mask tokens in the sentence or cells in the table (2) salient mention masking where we first identify phrases in the sentence that align with table elements, then mask aligned phrases (denoted as mentions). Compared to random masking, salient masking specifically focuses on masking shared information, enabling the model to better learn the correspondence across formats, which we will verify in § 4.3. Since we use TAPEX as the base model, which is based on BART, we follow the pretraining format of BART to generate the original unmasked sequence given the input with masked tokens (in either NL or table). Instead of computing the negative log likelihood loss (NLL) over all tokens, we only compute at masked positions to emphasize the recovery of missing information:
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
where s * /T * denote the masked sentence/table, and P mask (·|·) only computes loss at masked positions.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
We use synthetic questions to simulate real tablebased questions that involve various reasoning operations, such as the comparative operation in Fig. 1. While directly synthesizing complex NL questions is challenging, it is easier to generate complex structured queries such as SQL because they follow predefined syntax, and different reasoning operations can be implemented with different SQL templates. For example, the SQL query in Fig. 1 (2021) already sampled a large corpus of SQL with associated answers based on tables from Wikipedia and used SQL plus tables as input to pretrain their model TAPEX. They achieved state-of-the-art performance on tablebased QA, making TAPEX a strong base model. However, there is a large gap between SQL and NL questions, and training solely on SQL might hinder it from closing this gap. Instead, we use NL questions in the pretraining directly. Given synthesized NL questions, we pretrain with a standard generative QA loss that takes NL questions concatenated with tables as input and decode answers obtained by executing SQL queries over tables:
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Sampling SQL Based on Templates Liu et al. 
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Even with a strong model like BART, the accuracy of SQL2NL is not perfect, especially in the face of limited data. To further improve performance, we devise a verification-based self-training approach that selects high-quality NL questions generated from unlabeled SQL by assessing how likely they elicit correct answers from the table-QA model.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
As illustrated in Fig. 4, we first finetune BART model on supervised SQL-NL pairs to obtain the initial SQL2NL model (x), which is used to generate NL questions for unlabeled SQL (y) in the second step. We attempted two generation methods including beam search and top-k sampling (Fan et al., 2018) and found that beam search leads to more diverse results. Thus we use a beam size of 50 to generate candidate NL questionŝ q. Third, we choose high-quality candidates for self-training based on various criteria. The most straightforward criterion is to choose questions with the highest generation probabilities for selftraining score gen. (q) = P SQL2NL (q|o), which does not lead to improvement as we will show in the ablations ( § 4.3). Motivated by the fact that Om-niTab has a strong capacity to answer table-related questions after large-scale pretraining and finetuning, we rank generated sentences by verifying how likely they elicit the correct answer using OmniTab score ver. (q) = P (a|q, T ), which provides a simple and effective way to leverage the QA capacity of OmniTab to indirectly provide feedback to the SQL2NL model (z). Given the verification scores, we first pair each unlabeled SQL with the sentence with the highest score among 50 candidates, then keep the top-K SQL-NL ranked based on the score as our self-training data. At the last step, the small supervised data is combined with the enlarged selftraining data to train the final SQL2NL model ({).
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
As illustrated in Fig. 4, we first finetune BART model on supervised SQL-NL pairs to obtain the initial SQL2NL model (x), which is used to generate NL questions for unlabeled SQL (y) in the second step. We attempted two generation methods including beam search and top-k sampling (Fan et al., 2018) and found that beam search leads to more diverse results. Thus we use a beam size of 50 to generate candidate NL questionŝ q. Third, we choose high-quality candidates for self-training based on various criteria. The most straightforward criterion is to choose questions with the highest generation probabilities for selftraining score gen. (q) = P SQL2NL (q|o), which does not lead to improvement as we will show in the ablations ( § 4.3). Motivated by the fact that Om-niTab has a strong capacity to answer table-related questions after large-scale pretraining and finetuning, we rank generated sentences by verifying how likely they elicit the correct answer using OmniTab score ver. (q) = P (a|q, T ), which provides a simple and effective way to leverage the QA capacity of OmniTab to indirectly provide feedback to the SQL2NL model (z). Given the verification scores, we first pair each unlabeled SQL with the sentence with the highest score among 50 candidates, then keep the top-K SQL-NL ranked based on the score as our self-training data. At the last step, the small supervised data is combined with the enlarged selftraining data to train the final SQL2NL model ({).
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
We perform multitasking using the two previously defined objectives (Eq. 2, Eq. 3) to combine natural and synthetic data. In addition, since we use TAPEX as initialization and their SQL-based pretraining has demonstrated effective in endowing models with reasoning capability, we add SQLbased pretraining in the multitask setting. L sql = − log P (a|o, T ), resulting a combination of three parts L mask + L QA + L sql as the multitask loss.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
We first detail the experimental settings ( § 4.1). Then we report on extensive experiments, starting with the overall result including all design elements to achieve the best results ( § 4.2), then breaking down into each design choice to analyze their individual contribution ( § 4.3). Finally, we analyze concrete examples to provide insight on different characteristics of natural and synthetic data ( § 4.4).
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Few-shot Settings We use WikiTableQuestions (Pasupat and Liang, 2015) as our major benchmark, as it is a widely used table-based QA dataset involving various complex reasoning operations, and report the answer accuracy given by the official evaluator. Following work on few-shot QA (Ram et al., 2021), we create few-shot settings of WTQ by sampling a subset from the original training containing 11K examples in total, with sizes changing on a logarithmic scale from 16 to 1024.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Few-shot Settings We use WikiTableQuestions (Pasupat and Liang, 2015) as our major benchmark, as it is a widely used table-based QA dataset involving various complex reasoning operations, and report the answer accuracy given by the official evaluator. Following work on few-shot QA (Ram et al., 2021), we create few-shot settings of WTQ by sampling a subset from the original training containing 11K examples in total, with sizes changing on a logarithmic scale from 16 to 1024.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Another component that requires annotated NL questions is the SQL2NL model. We use SQUALL (Shi et al., 2020), which contains ≈10K annotations in total, to simulate few-shot scenarios by varying the number of SQL annotated with NL from 16 to 4,096. In the f -shot setting, we use SQUALL-f to train the SQL2NL model and WTQf to finetune QA models. Since SQUALL and WTQ share the same set of NL questions, we make sure that SQUALL-f also includes the same questions as WTQ-f , so in total only f annotated questions are used in the f -shot setting.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Another component that requires annotated NL questions is the SQL2NL model. We use SQUALL (Shi et al., 2020), which contains ≈10K annotations in total, to simulate few-shot scenarios by varying the number of SQL annotated with NL from 16 to 4,096. In the f -shot setting, we use SQUALL-f to train the SQL2NL model and WTQf to finetune QA models. Since SQUALL and WTQ share the same set of NL questions, we make sure that SQUALL-f also includes the same questions as WTQ-f , so in total only f annotated questions are used in the f -shot setting.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
We also report on WikiSQL (Zhong et al., 2017), another table-based QA benchmark with relatively simple questions. To evaluate robustness under topical shift, we further use WikiTableQuestions-TS which split WTQ into five topics (Chemmengath et al., 2021) based on Wikipedia categories. We follow their creation procedure to reproduce the split, and evaluate our methods by finetuning on one topic and testing on the other four topics.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Pretraining Corpora We use the table corpus by Herzig et al. (2020) extracted from Wikipedia as our source of tables for retrieval. All tables are preprocessed into a two-dimensional structure with a single header and one or multiple data rows. We use a subset of this corpus and find the corresponding Wikipedia page through its URL, which is preprocessed into sentences using SLING. Since some tables are noisy and some Wikipedia pages do not contain meaningful sentences, eventually we pair approximately 0.5M tables with sentences using our three retrieval methods. To make the synthetic data of similar size, we also use 0.5M SQL sampled by Liu et al. (2021) to generate synthetic questions.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Pretraining Corpora We use the table corpus by Herzig et al. (2020) extracted from Wikipedia as our source of tables for retrieval. All tables are preprocessed into a two-dimensional structure with a single header and one or multiple data rows. We use a subset of this corpus and find the corresponding Wikipedia page through its URL, which is preprocessed into sentences using SLING. Since some tables are noisy and some Wikipedia pages do not contain meaningful sentences, eventually we pair approximately 0.5M tables with sentences using our three retrieval methods. To make the synthetic data of similar size, we also use 0.5M SQL sampled by Liu et al. (2021) to generate synthetic questions.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Baselines We consider two types of models as baselines (1) pipeline methods that execute generated SQL to get answers such as TaBERT (Yin et al., 2020) with MAPO (Liang et al., 2018) as the semantic parser and (2) end2end methods that directly generate answers, such as BART (Lewis et al., 2020b), TAPAS (Herzig et al., 2020), and TAPEX (Liu et al., 2021). More discussions about table-related models can be found in the related work section in § 5. Since we also incorporate the SQL data used by TAPEX in our final multitask setting, we report TAPEX * when comparing with our multitask setting, which continued to train TAPEX on SQL data for as many steps as OmniTab to make a fair and rigorous comparison. We use OmniTab to denote our full model trained in the multitask setting with both natural, synthetic, and SQL data ( § 3.3).
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Baselines We consider two types of models as baselines (1) pipeline methods that execute generated SQL to get answers such as TaBERT (Yin et al., 2020) with MAPO (Liang et al., 2018) as the semantic parser and (2) end2end methods that directly generate answers, such as BART (Lewis et al., 2020b), TAPAS (Herzig et al., 2020), and TAPEX (Liu et al., 2021). More discussions about table-related models can be found in the related work section in § 5. Since we also incorporate the SQL data used by TAPEX in our final multitask setting, we report TAPEX * when comparing with our multitask setting, which continued to train TAPEX on SQL data for as many steps as OmniTab to make a fair and rigorous comparison. We use OmniTab to denote our full model trained in the multitask setting with both natural, synthetic, and SQL data ( § 3.3).
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Implementation Details During pretraining, we use a batch size of 512 and train OmniTab for 5 epochs, which takes about 20 hours on 8 V100 GPUs for multitasking on both natural and synthetic data. During finetuning, we use a batch size of 96 and finetune OmniTab for 50 epochs, which takes about 30 minutes on 8 V100 GPUs. We use a learning rate of 2e-5 for both pretraining, finetuning. We use BART-large and TAPEX-large in all experiments. For dense retrieval, since we use the max operation, all phrases have scores ∈ [−1, 1]. We bucket phrases into bins with an interval of 0.1, manually inspect the quality of some randomly sampled phrases from each bin, and found that phrases with scores larger than τ = 0.6 are of high quality. We use spaCy 4 to detect named entities for dense retrieval. For self-training of the SQL2NL model, we use the best-performing OmniTab model without self-training as the verification QA model, and make sure that it uses the same amount of annotations as the final model (i.e. if the final model is a f -shot model, we also use f annotations to train the verification model). In our final model, we added approximately 10K SQL-NL pairs for self-training. Pipeline systems TaBERT+MAPO (Yin et al., 2020) 7.7 15.1 33.3 52.3
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Implementation Details During pretraining, we use a batch size of 512 and train OmniTab for 5 epochs, which takes about 20 hours on 8 V100 GPUs for multitasking on both natural and synthetic data. During finetuning, we use a batch size of 96 and finetune OmniTab for 50 epochs, which takes about 30 minutes on 8 V100 GPUs. We use a learning rate of 2e-5 for both pretraining, finetuning. We use BART-large and TAPEX-large in all experiments. For dense retrieval, since we use the max operation, all phrases have scores ∈ [−1, 1]. We bucket phrases into bins with an interval of 0.1, manually inspect the quality of some randomly sampled phrases from each bin, and found that phrases with scores larger than τ = 0.6 are of high quality. We use spaCy 4 to detect named entities for dense retrieval. For self-training of the SQL2NL model, we use the best-performing OmniTab model without self-training as the verification QA model, and make sure that it uses the same amount of annotations as the final model (i.e. if the final model is a f -shot model, we also use f annotations to train the verification model). In our final model, we added approximately 10K SQL-NL pairs for self-training. Pipeline systems TaBERT+MAPO (Yin et al., 2020) 7.7 15.1 33.3 52.3
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
End2end systems BART (Lewis et al., 2020b) 2.9 8.4 17.3 38.0 TAPAS (Herzig et al., 2020) 9.8 18.9 33.6 48.8 TAPEX (Liu et al., 2021) 10  
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
End2end systems BART (Lewis et al., 2020b) 2.9 8.4 17.3 38.0 TAPAS (Herzig et al., 2020) 9.8 18.9 33.6 48.8 TAPEX (Liu et al., 2021) 10  
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
The overall results comparing OmniTab with other baselines are listed in Tab. 1. Across three fewshot settings, simulating low, medium, and high resource scenarios, pretraining on natural or synthetic data individually both outperform baselines significantly, and multitasking further increases the performance by a large margin. OmniTab improves over the best baseline performance by 11.1%, 16.2%, and 6.4% across the three settings, clearly demonstrating that pretraining on natural sentences relevant to tables and synthetic questions provides OmniTab with a stronger capacity to align text and tables and perform reasoning. The two types of data are complementary to each other, which we will analyze in detail in § 4.4. Despite the fact that we focus on the few-shot setting, we also observe significant improvements of 2.7% on the full setting, establishing a new state-of-the-art on WTQ. The performance in all few-shot/full settings shown in Fig. 5 clearly indicates the superiority of OmniTab across the whole spectrum. The improvement is larger when annotations are fewer, indicating the value of pretraining especially when fewer annotations are available. We also observe improvements on WikiSQL as shown in Tab  reinforcing the effectiveness of our methods.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Next, we study the contribution of individual components, including different retrieval methods, masking strategies, self-training methods, and varying the number of training pairs.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Our first ablation concerns the influence of different retrieval methods on the final performance. We examined three retrieval methods to pair tables with a relevant sentence, including string-based matching, BM25, and dense retrieval ( § 3.1), as summarized in Tab. 3. We also add a baseline (title-based heuristic) that pairs a table with the caption, article title, and description used by Herzig et al. (2020) to validate the utility of retrieval.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Our first ablation concerns the influence of different retrieval methods on the final performance. We examined three retrieval methods to pair tables with a relevant sentence, including string-based matching, BM25, and dense retrieval ( § 3.1), as summarized in Tab. 3. We also add a baseline (title-based heuristic) that pairs a table with the caption, article title, and description used by Herzig et al. (2020) to validate the utility of retrieval.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
(1) Our three retrieval methods usually perform better than the title-based heuristic, indicating that retrieving sentences based on the table content is better than fixed heuristics that always pair a table with pre-specified content.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
(2) By comparing two string-based matching variants, we found that selecting the sentence with the maximal number of mentions is better than sentences with minimal overlap, 5 confirming our intuition that sentences more aligned with the table enable models to learn better alignment.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
(3) BM25 performs similarly to string-based matching, partially because we still rely on string-based matching to locate mentions after BM25 returns a similar sentence. (4) Dense retrieval with threshold τ = 0.6 achieves the best overall performance, but it is relatively sensitive to the threshold. A high threshold only keeps highly relevant phrase-cell pairs, while a low threshold can discover more partial matches for masked pretraining, leading to a trade-off between quality and quantity. Given that this initial attempt to use dense retrieval for text-  tions of cells in the sentence and random masking in our final model. To examine the contribution of each masking strategy, we remove one masking strategy from the underlined model at the bottom of Tab. 3. It is clear that both maskings help, with salient masking being the major contributor, which indicates that masking tokens indicative of alignment is more effective than aimless masking.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
To study which element is crucial in self-training, we perform ablations to study various aspects of self-training including (1) selection criterion for questions (generation-vs verification-based) and
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
(2) models used for verification (BART vs Om-niTab) by comparing all variants under the same setting of 128 annotated SQL-NL. As summarized in Tab. 4, self-training on selected questions with the highest generation probabilities given by the SQL2NL model does not improve over the baseline without self-training, which is mainly because the SQL2NL model is too weak to output reliable generation probabilities. However, our method that selects questions with the highest probabilities to elicit answers from OmniTab (last line) improve over no self-training by a large margin (4.3%, 2.5%, and 1.8%), validating the idea of leveraging the strong QA capacity of OmniTab to assess the quality of generated questions. To confirm the source of success, we perform a sanity check that selects sentences most unlikely to elicit answers (min), and the performance indeed becomes much lower. We also replace OmniTab with BART that is only finetuned with 128 annotations, and the performance is significantly lower, confirming that stronger QA  Accuracy (%)
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Self-training Annotated Figure 6: WTQ test accuracy (128-shot) using different numbers of annotated and self-training SQL-NL pairs. models can provide a better assessment.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Performance w.r.t. Number of Annotated and Self-training Pairs Here we study the influence of increasing either annotated or self-training SQL-NL pairs. We use the SQL2NL model trained with 128 annotated pairs as a starting point, and additionally using more annotated or self-training pairs. As shown in Fig. 6, using more annotated or selftraining pairs both improves over the initial performance of 35.0%. However, the improvement due to self-training still falls far behind the supervised approach, demonstrating the challenge of learning a robust SQL2NL model with very few annotations. The increasing trend of self-training suggests that further improvements may be provided by using more pairs in self-training.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Roles of Natural and Synthetic Data We quantitatively verified using the multitasking experiment that natural and synthetic are complementary to
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Cases favoring: Natural Synthetic
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Avg #tok in questions / SQL 10.6 / 11.4 10.8 / 12.9 Avg #aligned question tok with tables 1.8 1.0 each other, with the hypothetical reason that natural data excels at enhancing alignment while synthetic data is more targeted on endowing reasoning capabilities. Our analysis on cases where natural pretraining succeeds but synthetic fails and the opposite cases confirms that this is indeed the case. Enabled by the fine-grained annotation in SQUALL (Shi et al., 2020), we compare OmniTab trained on natural or synthetic data separately in the 128-shot setting, and study on the two groups of cases in the development set of WikiTableQuestions. Based on 309/315 cases favoring natural/synthetic pretraining, we witness a clear distinction on the average number of question tokens aligned with tables between the two groups in Tab. 5, indicating that natural data is indeed more targeted at addressing the alignment across formats. We also compute the frequency of each SQL keyword for the two contrasting groups of cases. As shown in Fig. 7, cases favoring synthetic data indeed involves reasoningrich keywords more frequently, such as "where = ( )" which are often used in nested queries, and "count * id" which are often used in aggregation.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Avg #tok in questions / SQL 10.6 / 11.4 10.8 / 12.9 Avg #aligned question tok with tables 1.8 1.0 each other, with the hypothetical reason that natural data excels at enhancing alignment while synthetic data is more targeted on endowing reasoning capabilities. Our analysis on cases where natural pretraining succeeds but synthetic fails and the opposite cases confirms that this is indeed the case. Enabled by the fine-grained annotation in SQUALL (Shi et al., 2020), we compare OmniTab trained on natural or synthetic data separately in the 128-shot setting, and study on the two groups of cases in the development set of WikiTableQuestions. Based on 309/315 cases favoring natural/synthetic pretraining, we witness a clear distinction on the average number of question tokens aligned with tables between the two groups in Tab. 5, indicating that natural data is indeed more targeted at addressing the alignment across formats. We also compute the frequency of each SQL keyword for the two contrasting groups of cases. As shown in Fig. 7, cases favoring synthetic data indeed involves reasoningrich keywords more frequently, such as "where = ( )" which are often used in nested queries, and "count * id" which are often used in aggregation.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Last, we analyze the robustness of OmniTab under topical distributional shift on WTQ-TS, which splits WTQ into five topics. We finetune OmniTab on one topic (128-shot) and test the resulting model on all five topics. As indicated by Tab. 6, OmniTab outperforms TAPEX by a large margin across all topics, validating the robustness of our methods under topical shift. Table-based QA is a well-studied area from early systems using structured queries as intermediate steps (Krishnamurthy et al., 2017;Liang et al., 2018;Wang et al., 2019;Yin et al., 2020;Yu et al., 2021) to recent advances that generate answers in an end2end fashion (Herzig et al., 2020; (Deng et al., 2021; and table understanding (Deng et al., 2020; start witnessing efforts invested in pretraining on both structured and unstructured information. Most works leveraging retrieval to find relevant information to assist pretraining are designed for text format only (Guu et al., 2020;Lewis et al., 2020a), while the majority of table-based pretraining still use alignment heuristics (Herzig et al., 2020;Yin et al., 2020). There are some initial attempts to perform retrieval over tables (Oguz et al., 2020;Herzig et al., 2021;Ma et al., 2021), but they mainly use tables as an additional information source while we focus on pairing tables with text for pretraining.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Last, we analyze the robustness of OmniTab under topical distributional shift on WTQ-TS, which splits WTQ into five topics. We finetune OmniTab on one topic (128-shot) and test the resulting model on all five topics. As indicated by Tab. 6, OmniTab outperforms TAPEX by a large margin across all topics, validating the robustness of our methods under topical shift. Table-based QA is a well-studied area from early systems using structured queries as intermediate steps (Krishnamurthy et al., 2017;Liang et al., 2018;Wang et al., 2019;Yin et al., 2020;Yu et al., 2021) to recent advances that generate answers in an end2end fashion (Herzig et al., 2020; (Deng et al., 2021; and table understanding (Deng et al., 2020; start witnessing efforts invested in pretraining on both structured and unstructured information. Most works leveraging retrieval to find relevant information to assist pretraining are designed for text format only (Guu et al., 2020;Lewis et al., 2020a), while the majority of table-based pretraining still use alignment heuristics (Herzig et al., 2020;Yin et al., 2020). There are some initial attempts to perform retrieval over tables (Oguz et al., 2020;Herzig et al., 2021;Ma et al., 2021), but they mainly use tables as an additional information source while we focus on pairing tables with text for pretraining.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
We propose an omnivorous pretraining approach that consumes both natural and synthetic data to enhance the ability to understand and align text and tables and the ability to perform reasoning. Our extensive results demonstrate the effectiveness of both data and verify their complementary value.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
Our empirical results together with the case analysis indicate that omnivorous pretraining can indeed benefit from the merits of both data, encouraging future advances in retrieval and synthesis to obtain higher-quality data and better pretraining strategies to combine heterogeneous data.
Understanding tables with intermediate pre-training
Textual entailment (Dagan et al., 2005), also known as natural language inference (Bowman et al., 2015), is a core natural language processing (NLP) task. It can predict effectiveness of reading comprehension (Dagan et al., 2010), which argues that it can form the foundation of many other NLP tasks, and is a useful neural pre-training task (Subramanian et al., 2018;Conneau et al., 2017).
Understanding tables with intermediate pre-training
Textual entailment is well studied, but many relevant data sources are structured or semi-structured: health data both worldwide and personal, fitness trackers, stock markets, and sport statistics. While some information needs can be anticipated by handcrafted templates, user queries are often surprising, and having models that can reason and parse that structure can have a great impact in real world applications (Khashabi et al., 2016;Clark, 2019).
Understanding tables with intermediate pre-training
A recent example is TABFACT , a dataset of statements that are either entailed or refuted by tables from Wikipedia (Figure 1). Because solving these entailment problems requires sophisticated reasoning and higher-order operations like arg max, averaging, or comparing, human accuracy remains substantially (18 points) ahead of the best models (Zhong et al., 2020).
Understanding tables with intermediate pre-training
The current models are dominated by semantic parsing approaches that attempt to create logical forms from weak supervision. We, on the other hand, follow Herzig et al. (2020) and  and encode the tables with BERT-based models to directly predict the entailment decision. But while BERT models for text have been scrutinized and optimized for how to best pre-train and represent textual data, the same attention has not been applied to tabular data, limiting the effectiveness in this setting. This paper addresses these shortcomings using intermediate task pretraining (Pruksachatkun et al., 2020), creating efficient data representations, and applying these improvements to the tabular entailment task.
Understanding tables with intermediate pre-training
Our methods are tested on the English language, mainly due to the availability of the end task resources. However, we believe that the proposed solutions could be applied in other languages where a pre-training corpus of text and tables is available, such as the Wikipedia datasets.
Understanding tables with intermediate pre-training
Our main contributions are the following: i) We introduce two intermediate pre-training tasks, which are learned from a trained MASK-LM model, one based on synthetic and the other from counterfactual statements. The first one generates a sentence by sampling from a set of logical expressions that filter, combine and compare the information on the table, which is required in table entailment (e.g., knowing that Gerald Ford is taller than the average president requires summing all presidents and dividing by the number of presidents). The second one corrupts sentences about tables appearing on Wikipedia by swapping entities for plausible alternatives. Examples of the two tasks can be seen in Figure 1. The procedure is described in detail in section 3.
Understanding tables with intermediate pre-training
ii) We demonstrate column pruning to be an effective means of lowering computational cost at minor drops in accuracy, doubling the inference speed at the cost of less than one accuracy point.
Understanding tables with intermediate pre-training
iii) Using the pre-training tasks, we set the new state-of-the-art on TABFACT out-performing previous models by 6 points when using a BERT-base model and 9 points for a BERT-large model. The procedure is data efficient and can get comparable accuracies to previous approaches when using only 10% of the data. We perform a detailed analysis of the improvements in Section 6. Finally, we show that our method improves the state-of-the-art on a question answering task (SQA) by 4 points.
Understanding tables with intermediate pre-training
We release the pre-training checkpoints, data generation and training code at github.com/googleresearch/tapas.
Understanding tables with intermediate pre-training
We use a model architecture derived from BERT and add additional embeddings to encode the table structure, following the approach of Herzig et al. (2020) to encode the input.
Understanding tables with intermediate pre-training
The statement and table in a pair are tokenized into word pieces and concatenated using the standard [CLS] and [SEP] tokens in between. The table is flattened row by row and no additional separator is added between the cells or rows. Six types of learnable input embeddings are added together as shown in Appendix B. Token embeddings, position embeddings and segment embeddings are analogous to the ones used in standard BERT. Additionally we follow Herzig et al. (2020) and use column and row embeddings which encode the two dimensional position of the cell that the token corresponds to and rank embeddings for numeric columns that encode the numeric rank of the cell with respect to the column, and provide a simple way for the model to know how a row is ranked according to a specific column.
Understanding tables with intermediate pre-training
Recall that the bi-directional self-attention mechanism in transformers is unaware of order, which motivates the usage of positional and segment embeddings for text in BERT, and generalizes naturally to column and row embeddings when processing tables, in the 2-dimensional case.
Understanding tables with intermediate pre-training
Let s and T represent the sentence and table respectively and E s and E T be their corresponding input embeddings. The sequence (Vaswani et al., 2017) denoted f and a contextual representation is obtained for every token. We model the probability of entailment P (s|T ) with a single hidden layer neural network computed from the output of the [CLS] token:
Understanding tables with intermediate pre-training
where the middle layer has the same size as the hidden dimension and uses a tanh activation and the final layer uses a sigmoid activation.
Understanding tables with intermediate pre-training
The use of challenging pre-training tasks has been successful in improving downstream accuracy (Clark et al., 2020). One clear caveat of the method adopted in Herzig et al. (2020) which attempts to fill in the blanks of sentences and cells in the table is that not much understanding of the table in relation with the sentence is needed.
Understanding tables with intermediate pre-training
With that in mind, we propose two tasks that require sentence-table reasoning and feature complex operations performed on the table and entities grounded in sentences in non-trivial forms.
Understanding tables with intermediate pre-training
We discuss two methods to create pre-training data that lead to stronger table entailment models. Both methods create statements for existing Wikipedia tables 2 . We extract all tables that have at least two columns, a header row and two data rows. We recursively split tables row-wise into the upper and lower half until they have at most 50 cells. This way we obtain 3.7 million tables.
Understanding tables with intermediate pre-training
Motivated by work on counterfactually-augmented data (Kaushik et al., 2020;Gardner et al., 2020), we propose an automated and scalable method to get table entailments from Wikipedia and, for each such positive examples, create a minimally differing refuted example. For this pair to be useful we want that their truth value can be predicted from the associated table but not without it.
Understanding tables with intermediate pre-training
The tables and sentences are extracted from Wikipedia as follows: We use the page title, description, section title, text and caption. We also use all sentences on Wikipedia that link to the table's page and mentions at least one page (entity) that is also mentioned in the table. Then these snippets are split into sentences using the NLTK (Loper and Bird, 2002) implementation of Punkt (Kiss and Strunk, 2006). For each relevant sentence we create one positive and one negative statement.
Understanding tables with intermediate pre-training
Consider the table in Figure 1 and the sentence '[Greg Norman] is [Australian].' (Square brackets indicate mention boundaries.). A mention 3 is a potential focus mention if the same entity or value is also mentioned in the table. In our example, Greg Norman and Australian are potential focus mentions. Given a focus mention (Greg Norman) we define all the mentions that occur in the same column (but do not refer to the same entity) as the replacement mentions (e.g., Billy Mayfair, Lee Janzen, . . . ). We expect to create a false statement if we replace the focus mention with a replacement mention (e.g., 'Billy Mayfair is Australian.'), but there is no guarantee it will be actually false.
Understanding tables with intermediate pre-training
We call a mention of an entity that occurs in the same row as the focus entity a supporting mention, because it increases the chance that we falsify the statement by replacing the focus entity. In our example, Australian would be a supporting mention for Greg Norman (and vice versa). If we find a supporting mention we restrict the replacement candidates to the ones that have a different value.
Understanding tables with intermediate pre-training
In the example, we would not use Steve Elkington since his row also refers to Australia.
Understanding tables with intermediate pre-training
Some replacements can lead to ungrammatical statements that a model could use to identify the negative statements, so we found it is useful to also replace the entity in the original positive sentence from Wikipedia with the mention from the table. 4 We also introduce a simple type system for entities (named entity, date, cardinal number and ordinal number) and only replace entities of the same type. Short sentences having less than 4 tokens not counting the mention, are filtered out.
Understanding tables with intermediate pre-training
Using this approach we extract 4.1 million counterfactual pairs of which 546 thousand do have a supporting mention and the remaining do not.
Understanding tables with intermediate pre-training
We evaluated 100 random examples manually and found that the percentage of negative statements that are false and can be refuted by the table is 82% when they have a supporting mention and 22% otherwise. Despite this low value we still found the examples without supporting mention to improve accuracy on the end tasks (Appendix F).
Understanding tables with intermediate pre-training
Motivated by previous work (Geva et al., 2020), we propose a synthetic data generation method to improve the handling of numerical operations and comparisons. We build a table-dependent statement that compares two simplified SQL-like expressions. We define the (probabilistic) context-free grammar shown in Figure 2. Synthetic statements are sampled from the CFG. We constrain the select values of the left and right expression to be either both the count or to have the same value for column .   Table 1.
Understanding tables with intermediate pre-training
Result first the value in C with the lowest row index. last the value in C with the highest row index. greatest the value in C with the highest numeric value. lowest the value in C with the lowest numeric value. sum
Understanding tables with intermediate pre-training
The sum of all the numeric values. average The average of all the numeric values. range
Understanding tables with intermediate pre-training
The difference between greatest and lowest. This guarantees that the domains of both expressions are comparable. value is chosen as at random from the respective column. A statement is redrawn if it yields an error (see Table 1). With probability 0.5 we replace one of both expressions by the values it evaluates to. In the example given in figure 1 
Understanding tables with intermediate pre-training
Some input examples from TABFACT can be too long for BERT-based models. We evaluate table pruning techniques as a pre-processing step to select relevant columns that respect the input example length limits. As described in section 2, an example is built by concatenating the statement with the flattened table. For large tables the example length can exceed the capacity limit of the transformer.
Understanding tables with intermediate pre-training
The TAPAS model handles this by shrinking the text in cells. A token selection algorithm loops over the cells. For each cell it starts by selecting the first token, then the second and so on until the maximal length is reached. Unless stated otherwise we use the same approach. Crucially, selecting only relevant columns would allow longer examples to fit without discarding potentially relevant tokens.
Understanding tables with intermediate pre-training
Heuristic entity linking (HEL) is used as a baseline. It is the table pruning used in TABLE-BERT . The algorithm aligns spans in statement to the columns by extracting the longest character n-gram that matches a cell. The span matches represent linked entities. Each entity in the statement can be linked to only one column. We use the provided entity linking statements data 5 . We run the TAPAS algorithm on top of the input data to limit the input size.
Understanding tables with intermediate pre-training
We propose a different method that tries to retain as many columns as possible. In our method, the columns are ranked by a relevance score and added in order of decreasing relevance. Columns that exceed the maximum input length are skipped. The algorithm is detailed in Appendix F. Heuristic exact match (HEM) computes the Jaccard coefficient between the statement and each column. Let T S be the set of tokens in the statement S and T C the tokens in column C, with C ∈ C the set of columns. Then the score between the statement and column is given by |T S ∩T C | |T S ∪T C | . We also experimented with approaches based on word2vec (Mikolov et al., 2013), character overlap and TF-IDF. Generally, they produced worse results than HEM. Details are shown in Appendix F.
Understanding tables with intermediate pre-training
In all experiments, we start with the public TAPAS checkpoint, 6 train an entailment model on the data from Section 3 and then fine-tune on the end task (TABFACT or SQA). We report the median accuracy values over 3 pre-training and 3 fine-tuning runs (9 runs in total). We estimate the error margin as half the interquartile range, that is half the difference between the 25th and 75th percentiles. The hyper-parameters, how we chose them, hardware and other information to reproduce our experiments are explained in detail in Appendix A.
Understanding tables with intermediate pre-training
The training time depends on the sequence length used. For a BERT-Base model it takes around 78 minutes using 128 tokens and it scales almost linearly up to 512. For our pre-training tasks, we explore multiple lengths and how they trade-off speed for downstream results.
Understanding tables with intermediate pre-training
We evaluate our model on the recently released TABFACT dataset . The tables are extracted from Wikipedia and the sentences written by crowd workers in two batches. The first batch consisted of simple sentences, that instructed the writers to refer to a single row in the table. The second one, created complex sentences by asking writers to use information from multiple rows.
Understanding tables with intermediate pre-training
In both cases, crowd workers initially created only positive (entailed) pairs, and in a subsequent annotation job, the sentences were copied and edited into negative ones, with instructions of avoiding simple negations. Finally, there was a third verification step to filter out bad rewrites. The final count is 118, 000. The split sizes are given in Appendix C. An example of a table and the sentences is shown in Figure 1. We use the standard TABFACT split and the official accuracy metric.
Understanding tables with intermediate pre-training
We also use the SQA (Iyyer et al., 2017) dataset for pre-training (following Herzig et al. (2020)) and for testing if our pre-training is useful for related tasks. SQA is a question answering dataset that was created by asking crowd workers to split a compositional subset of WikiTableQuestions (Pasupat and Liang, 2015) into multiple referential questions. The dataset consists of 6,066 sequences (2.9 question per sequence on average). We use the standard split and official evaluation script.
Understanding tables with intermediate pre-training
Chen et al. (2020) present two models, TABLE-BERT and the Latent Program Algorithm (LPA), that yield similar accuracy on the TABFACT data.
Understanding tables with intermediate pre-training
LPA tries to predict a latent program that is then executed to verify if the statement is correct or false. The search over programs is restricted using lexical heuristics. Each program and sentence is encoded with an independent transformer model and then a linear layer gives a relevance score to the pair. The model is trained with weak supervision where programs that give the correct binary answer are considered positive and the rest negative.
Understanding tables with intermediate pre-training
TABLE-BERT is a BERT-base model that similar to our approach directly predicts the truth value of the statement. However, the model does not use special embeddings to encode the table structure but relies on a template approach to format the table as natural language. The table is mapped into a single sequence of the form: "Row 1 Rank is 1; the Player is Greg Norman; ... . Row 2 ...". The model is also not pre-trained on table data.
Understanding tables with intermediate pre-training
LOGICALFACTCHECKER (Zhong et al., 2020) is another transformer-based model that given a candidate logical expression, combines contextual embeddings of program, sentence and table, with a tree- RNN (Socher et al., 2013) to encode the parse tree of the expression. The programs are obtained through either LPA or an LSTM generator (Seq2Action).
Understanding tables with intermediate pre-training
TABFACT In Table 2 we find that our approach outperforms the previous state-of-the-art on TAB-FACT by more than 6 points (Base) or more than 9 points (Large). A model initialized only with the public TAPAS MASK-LM checkpoint is behind state-of-the-art by 2 points (71.7% vs 69.9%). If we train on the counterfactual data, it out-performs LOGICALFACTCHECKER and reaches 75.2% test accuracy (+5.3), slightly above using SQA. Only using the synthetic data is better (77.9%), and when using both datasets it achieves 78.5%. Switching from BERT-Base to Large improves the accuracy by another 2.5 points. The improvements are consistent across all test sets.
Understanding tables with intermediate pre-training
The pre-trained models are in principle already complete table entailment predictors. Therefore it is interesting to look at their accuracy on the TABFACT evaluation set before fine-tuning them. We find that the best model trained on all the pretraining data is only two points behind the fully trained TABLE-BERT (63.8% vs 66.1%). This relatively good accuracy mostly stems from the counterfactual data.
Understanding tables with intermediate pre-training
When looking at low data regimes in Figure  3 we find that pre-training on SQA or our artificial data consistently leads to better results than just training with the MASK-LM objective. The models with synthetic pre-training data start outperforming TABLE-BERT when using 5% of the    (2020) and Zhong et al. (2020). The best BERT-base model while comparable in parameters out-performs TABLE-BERT by more than 12 points. Pre-training with counterfactual and synthetic data gives an accuracy 8 points higher than only using MASK-LM and more than 3 points higher than using SQA. Both counterfactual and synthetic data out-perform pre-training with a MASK-LM objective and SQA. Joining the two datasets gives an additional improvement. Error margins are estimated as half the interquartile range.    training set. The setup with all the data is consistently better than the others and synthetic and counterfactual are both better than SQA.
Understanding tables with intermediate pre-training
SQA Our pre-training data also improves the accuracy on a QA task. On SQA (Iyyer et al., 2017) a model pre-trained on the synthetic entailment data outperforms one pre-trained on the MASK-LM task alone (Table 3) Table 3: SQA test results. ALL is the average question accuracy and SEQ the sequence accuracy. Both counterfactual and synthetic data out-perform the MASK-LM objective. Our Large model outperforms the MASK-LM model by almost 4 points on both metrics. Our best Base model is comparable to the previous state-of-the-art. Error margins are estimated as half the interquartile range.
Understanding tables with intermediate pre-training
Efficiency As discussed in Section 3.3 and Appendix A.4, we can increase the model efficiency by reducing the input length. By pruning the input of the TABFACT data we can improve training as well as inference time. We compare pruning with the heuristic entity linking (HEL)  and heuristic exact match (HEM) to different target lengths. We also studied other pruning methods, the results are reported in Appendix F. In Table 4 we find that HEM consistently outperforms HEL. The best model at length 256, while twice as fast to train (and apply), is only 0.8 points behind the best full length model. Even the model    with length 128, while using a much shorter length, out-performs TABLE-BERT by more than 7 points. Given a pre-trained MASK-LM model our training consists of training on the artificial pre-training data and then fine-tuning on TABFACT. We can therefore improve the training time by pre-training with shorter input sizes. Table 4 shows that 512 and 256 give similar accuracy while the results for 128 are about 1 point lower.
Understanding tables with intermediate pre-training
Salient Groups To obtain detailed information of the improvements of our approach, we manually annotated 200 random examples with the complex operations needed to answer them. We found 4 salient groups: Aggregations, superlatives, com-7 Not explicitly mentioned in the paper but implied by the batch size given (6) and the defaults in the code. paratives and negations, and sort pairs into these groups via keywords in the text. To make the groups exclusive, we add a fifth case when more than one operation is needed. The accuracy of the heuristics was validated through further manual inspection of 50 samples per group. The trigger words of each group are described in Appendix G.
Understanding tables with intermediate pre-training
For each group within the validation set, we look at the difference in accuracy between different models. We also look at how the total error rate is divided among the groups as a way to guide the focus on pre-training tasks and modeling. The error rate defined in this way measures potential accuracy gains if all the errors in a group S were fixed: ER(S) = |{ Errors in S}| |{ Validation examples}| . Among the groups, the intermediate task data improve superlatives (39% error reduction) and negations (31%) most (Table 5). For example, we see that the accuracy is higher for superlatives than the for the overall validation set.
Understanding tables with intermediate pre-training
In Figure 4 we show examples in every group where our model is correct on the majority of the cases (across 9 trials), and the MASK-LM baseline is not. We also show examples that continue to produce errors after our pre-training. Many examples in this last group require multi-hop reasoning or complex numerical operations.
Understanding tables with intermediate pre-training
Model Agreement Similar to other complex binary classification datasets such as BOOLQ (Clark et al., 2019), for TABFACT one may question whether models are guessing the right answer. To detect the magnitude of this issue we look at 9 independent runs of each variant and analyze how many of them agree on the correct answer. Figure 5 shows that while for MASK-LM only for 24.2% of the examples all models agree on the right answer, it goes up to 55.5% when using using the counterfactual and synthetic pre-training. This suggests that the amount of guessing decreases substantially.
Understanding tables with intermediate pre-training
Consistently
Understanding tables with intermediate pre-training
Choi Moon -Sik played in Seoul three times in total.
Understanding tables with intermediate pre-training
The total number of bronze medals were half of the total number of medals.
Understanding tables with intermediate pre-training
Mapiu school has the highest roll in the state authority.
Understanding tables with intermediate pre-training
Carlos Moya won the most tournaments with two wins. Comparatives Bernard Holsey has 3 more yards than Angel Rubio.
Understanding tables with intermediate pre-training
In 1982, the Kansas City Chiefs played more away games than home games.
Understanding tables with intermediate pre-training
The Warriors were not the home team at the game on 11-24-2006.
Understanding tables with intermediate pre-training
Dean Semmens is not one of the four players born after 1981. give the correct answer, out of 9 runs. Better pre-training leads to more consistency across models. The ratio of samples answered correctly by all models is 24.2% for MASK-LM but 55.5% for Synthetic + Counterfactual.
Understanding tables with intermediate pre-training
context amounts to detecting whether the table and the sentence appear in the same article, was not found to be effective. Our hypothesis is that the task was not hard enough to provide a training signal. We build on top of the TAPAS model and propose harder and more effective pre-training tasks to achieve strong performance on the TAB-FACT dataset.
Understanding tables with intermediate pre-training
Entailment tasks Recognizing entailment has a long history in NLP (Dagan et al., 2010). Recently, the text to text framework has been expanded to incorporate structured data, like knowledge graphs (Vlachos and Riedel, 2015), tables (Jo et al., 2019; or images (Suhr et al., 2017(Suhr et al., , 2019. The large-scale TABFACT dataset  is one such example. Among the top performing models in the task is a BERT based model, acting on a flattened versioned of the table using textual templates to make the tables resemble natural text. Our approach has two key improvements: the usage of special embeddings, as introduced in Herzig et al. (2020), and our novel counterfactual and synthetic pre-training (Section 3). Intermediate Pre-training Language model fine-tuning (Howard and Ruder, 2018) also know as domain adaptive pre-training (Gururangan et al., 2020) has been studied as a way to handle covariate shift. Our work is closer to intermediate task fine-tuning (Pruksachatkun et al., 2020) where one tries to teach the model higher-level abilities. Similarly we try to improve the discrete and numeric reasoning capabilities of the model.
Understanding tables with intermediate pre-training
The most similar approach to ours appears in Xiong et al. (2020), replacing entities in Wikipedia by others with the same type for a MASK-LM model objective. We, on the one hand, take advantage of other rows in the table to produce plausible negatives, and also replace dates and numbers. Recently, Kaushik et al. Numeric reasoning Numeric reasoning in Natural Language processing has been recognized as an important part in entailment models (Sammons et al., 2010) and reading comprehension (Ran et al., 2019). Wallace et al. (2019) studied the capacity of different models on understanding numerical operations and show that BERT-based model still have headroom. This motivates the use of the synthetic generation approach to improve numerical reasoning in our model.
Understanding tables with intermediate pre-training
Synthetic data generation Synthetic data has been used to improve learning in NLP tasks (Alberti et al., 2019;Lewis et al., 2019;Wu et al., 2016;Leonandya et al., 2019). In semantic parsing for example (Wang et al., 2015;Iyer et al., 2017;Weir et al., 2020), templates are used to bootstrap models that map text to logical forms or SQL. Salvatore et al. (2019) use synthetic data generated from logical forms to evaluate the performance of textual entailment models (e.g., BERT). Geiger et al. (2019) use synthetic data to create fair evaluation sets for natural language inference. Geva et al. (2020) show the importance of injecting numerical reasoning via generated data into the model to solve reading comprehension tasks. They propose different templates for generating synthetic numerical examples. In our work we use a method that is better suited for tables and to the entailment task, and is arguably simpler.
Understanding tables with intermediate pre-training
We introduced two pre-training tasks, counterfactual and synthetic, to obtain state-of-the-art results on the TABFACT (Chen et al., 2020) entailment task on tabular data. We adapted the BERT-based architecture of TAPAS (Herzig et al., 2020) to binary classification and showed that pre-training on both tasks yields substantial improvements on TAB-FACT but also on a QA dataset, SQA (Iyyer et al., 2017), even with only a subset of the training data.
Understanding tables with intermediate pre-training
We ran a study on column selection methods to speed-up training and inference. We found that we can speed up the model by a factor of 2 at a moderate drop in accuracy (≈ 1 point) and by a factor of 4 at a larger drop but still with higher accuracy than previous approaches.
Understanding tables with intermediate pre-training
We characterized the complex operations required for table entailment to guide future research in this topic. Our code and models will be opensourced. We provide details on our experimental setup and hyper-parameter tuning in Section A. Section B and C give additional information on model and the TABFACT dataset. We give details and results regarding our column pruning approach in Section D.
Understanding tables with intermediate pre-training
Full results for SQA are displayed in Section E. Section F shows the accuracy on the pre-training tasks held-out sets. Section G contains the trigger words used for identifying the salient groups in the analysis section.
Understanding tables with intermediate pre-training
The hyper-parameters are optimized using a black box Bayesian optimizer similar to Google Vizier (Golovin et al., 2017) which looked at validation accuracy after 8, 000 steps only, in order to prevent over-fitting and use resources effectively. The ranges used were a learning rate from 10 −6 to 3 × 10 −4 , dropout probabilities from 0 to 0.2 and warm-up ratio from 0 to 0.05. We used 200 runs and kept the median values for the top 20 trials. In order to show the impact of the number of trials in the expected validation results, we follow Henderson et al. (2018) and Dodge et al. (2019). Given that we used Bayesian optimization instead of random search, we applied the bootstrap method to estimate mean and variance of the max validation accuracy at 8, 000 steps for different number of trials. From trial 10 to 200 we noted an increase of 0.4% in accuracy and a standard deviation that decreases from 2% to 1.3%.
Understanding tables with intermediate pre-training
We use the same hyper-parameters for pre-training and fine-tuning. For pre-training, the input length is 256 and 512 for fine-tuning if not stated otherwise. We use 80, 000 training steps, a learning rate of 2e −5 and a warm-up ratio of 0.05. We disable the attention dropout in BERT but use a hidden dropout probability of 0.07 . Finally, we use an Adam optimizer with weight decay with the same configuration as BERT.
Understanding tables with intermediate pre-training
For SQA we do not use any search algorithm and use the same model and the same hyper-parameters as the ones used in Herzig et al. (2020). The only difference is that we start the fine-tuning from a checkpoint trained on our intermediate pre-training entailment task.  Figure 6: Input representation for model.
Understanding tables with intermediate pre-training
The number of parameters is the same as for BERT: 110M for base models and 340M for Large models.
Understanding tables with intermediate pre-training
We train all our models on Cloud TPUs V3. 
Understanding tables with intermediate pre-training
For illustrative purposes, we include the input representation using the 6 types of embeddings, as depicted by Herzig et al. (2020).
Understanding tables with intermediate pre-training
Statistics of the TABFACT dataset can be found in   
Understanding tables with intermediate pre-training
Let cost(.) ∈ N be the function that computes the number of tokens given a text using the BERT tokenizer, t s the tokenized statement text, t c i the text of the column i. We denote the columns as (c 1 , .., c n ) ordered by their scores
Understanding tables with intermediate pre-training
where n is the number of columns. Let m be the maximum number of tokens. Then the cost of the column must verify the following condition.
Understanding tables with intermediate pre-training
where C + i is the set of retained columns at the iteration i. 2 is added to the condition as two special tokens are added to the input:
Understanding tables with intermediate pre-training
[CLS], t s , [SEP ], tc 1 , ..., tc n . If a current column c i doesn't respect the condition then the column is not selected. Whether or not the column is retained, the algorithm continues and verifies if the next column can fit. It follows C +n contains the maximum number of columns that can fit under m by respecting the columns scoring order.
Understanding tables with intermediate pre-training
There is a number of heuristic pruning approaches we have experimented with. Results are given in 7. where v i represents the embedding of the token i. For a given column token c we define the relevance with respect to the statement as the average similarity to every token:
Understanding tables with intermediate pre-training
Where τ is a threshold that helps to remove noise from unrelated word embeddings. We set τ to 0.89. We experimented with max and sum as other aggregation function but found the average to perform best. The final score between the statement S and the column C is given by
Understanding tables with intermediate pre-training
Term frequencyâȂŞinverse document frequency (IWF) Scores the columns' tokens proportional to the word frequency in the statement and offset by the word frequency computed over all the tables and statements from the training set.
Understanding tables with intermediate pre-training
Where T F (t s , c) is how often the token c occurs in the statement t s , and W F (c) is the frequency of c in a word count list. The final score of a column C is given by
Understanding tables with intermediate pre-training
Character N-gram (CHAR) Scores columns by character overlap with the statement. This method looks for sub-list of wordâȂŹs characters in the statement. The length of the characters' list has a minimum and maximum length allowed. In the experiments we use 5 and 20 as minimum and maximum length. Let L s,c be the set of all the overlapping characters' lengths. The scoring for each column is given by f (t s , t c ) = min(max(L s,c , 5)), 20) cost(t c )  , Heuristic exact match (HEM), word-to-vec (W2V), inverse word frequency (IWF), character ngram (CHAR) at different pre-training (PT) and fine-tuning (FT) sizes. Error margins are estimated as half the interquartile range. Table 8 shows the accuracy on the first development fold and the test set. As for the main results, the error margins displayed are half the interquartile range over 9 runs, which is half the difference between the first and third quartile. This range contains half of the runs and provides a measure of robustness.
Understanding tables with intermediate pre-training
When training on the pre-training data we hold out approximately 1% of the data for testing how well the model is solving the pre-training task. In Table  9, we compare the test pre-training accuracy on synthetic and counterfactual data to models that are only trained on the statements to understand whether there is considerable bias in the data. Both datasets have some bias (i.e. the accuracy without table is higher than 50%.). Still there is a sufficient enough gap between training with and without tables so that the data is still useful. The synthetic data can be solved almost perfectly whereas for the counterfactual data we only reach an accuracy of 84.3%. This is expected as there is no guarantee that the model has enough information to decide whether a statement is true or false for the counterfactual examples.  Table 9: Accuracy on synthetic (Val S ) and counterfactual held-out sets (Val C ) of the pre-traininig data.
Understanding tables with intermediate pre-training
In table 10 we show the ablation results when removing the counterfactual statements that lack a supporting entity, that is a second entity that appears in both the table and sentence. This increases the probability that our generated negative pairs are incorrect, but it also discards 7 out of 8 examples, which ends up hurting the results. 
Understanding tables with intermediate pre-training
In table 11 we show the words that are used as markers to define each of the groups. We first identified manually the operations that were most often needed to solve the task and found relevant words linked with each group. The heuristic was validated by manually inspecting 50 samples from each group and observing higher than 90% accuracy.  Table 8: SQA dev (first fold) and test results. ALL is the average question accuracy, SEQ the sequence accuracy, and QX, the accuracy of the X'th question in a sequence. We show the median over 9 trials, and errors are estimated with half the interquartile range .
Understanding tables with intermediate pre-training
Aggregations total, count, average, sum, amount, there, only Superlatives first, highest, best, newest, most, greatest, latest, biggest and their opposites Comparatives than, less, more, better, worse, higher, lower, shorter, same Negations not, any, none, no, never 
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Pre-trained language models (LMs) such as BERT (Devlin et al., 2019) and BART (Lewis et al., 2020) have hit a success on a range of free-form natural language (NL) tasks. By learning from a large amount of unstructured textual data, these models have demonstrated surprising capabilities in understanding NL sentences. Inspired by this huge success, researchers have attempted to extend pre-training to structured tabular data (Herzig et al., 2020;Yin et al., 2020;Yu et al., 2021a;Wang et al., 2021b;Deng et al., 2020;Shi et al., 2021a). However, different from free-form NL sentences, tabular data often contains rich and meaningful structural information, for which existing pre-training approaches designed for unstructured data are not well suited.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
To apply pre-training techniques on structured tabular data, there exist two key challenges: (i) where to obtain a large-scale pre-training corpus with high quality, and (ii) how to design an efficient pretraining task for table pre-training. For the first challenge, existing works generally collect parallel data including NL sentences and tables as the pre-training corpus, since downstream tasks often involve a joint reasoning over both free-form NL sentences and tables. They either crawled tables and their surrounding NL sentences from the Web (Herzig et al., 2020;Yin et al., 2020;Deng et al., 2021), or synthesized NL sentences on available tables (Yu et al., 2021a;Shi et al., 2021a). However, as pointed by Yin et al. (2020), the raw data mined from the Web is extremely noisy and requires complicated heuristics to clean. Conversely, the synthesis method is easier to control the data quality, but it usually requires experts to write hundreds of templates, which is both costly and often lacking diversity. Regarding the pre-training task, existing works often employ different variants of Masked Language Modeling (MLM) (Devlin et al., 2019) to guide LMs to learn better representations of Input: Greece held its last Summer Olympics in which year? [Table] Output: 2004 Fine-tuning Fine-tuned LM for Downstream Task Figure 1: The schematic overview of our method. For the sake of brevity, the table content in the input is simplified with the symbol [Table]. tabular data. For example, TAPAS (Herzig et al., 2020) used MLM with whole word masking, and TABERT (Yin et al., 2020) proposed Masked Column Prediction (MCP) to encourage the model to recover the names and data types of masked columns. Despite their success, they still largely treat tabular data as a structural format of text, which leads to the need of an extremely large corpus for their table pre-training. All of these hinder the progress of table pre-training.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
In this paper, we present a novel execution-centric table pre-training approach TAPEX (TAble Pretraining via EXecution). It addresses the above challenges and achieves efficient table pre-training via approximating the structural reasoning process of formal languages over tables. The structural reasoning process is associated with the executability of tables, i.e., tables are inherently capable of supporting various reasoning operations (e.g., summing over a column in the table). In particular, TAPEX approximates the structural reasoning process of SQL queries by pre-training LMs to mimic the behavior of a SQL execution engine on tables. As shown in Figure 1, by sampling executable SQL queries over tables, TAPEX first synthesizes a large-scale pre-training corpus. Then it continues pre-training a language model to output the execution results of these SQL queries, which are obtained from the SQL execution engine. Since the diversity of SQL queries can be systematically guaranteed, we can easily synthesize a diverse, large-scale, and high-quality pre-training corpus.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Our key insight is that if a language model can be pre-trained to faithfully "execute" SQL queries and produce correct results, it should have a deep understanding of tables. Thus, the execution pre-training task could be more efficient in understanding tables and reasoning over tables. To our knowledge, TAPEX is the first one to explore table pre-training via synthetic executable programs.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
TAPEX is conceptually simple and easy to implement. In this paper, we regard the pre-training as a sequence generation task and employ an encoder-decoder model. Specifically, we employ the pre-trained encoder-decoder language model BART (Lewis et al., 2020) as the backbone. Furthermore, we examine the effectiveness of TAPEX via two fundamental downstream tasks: table-based question answering (TableQA) and table-based fact verification (TableFV). To enable fine-tuning of downstream tasks to take full advantage of TAPEX, we reformulate these tasks using the encoderdecoder sequence generation paradigm. We evaluate TAPEX using four well-known benchmark datasets. Experimental results clearly demonstrate that TAPEX can bring significant and consistent improvements on these datasets. For example, TAPEX obtains an absolute improvement of 19.5% over BART in the WIKITABLEQUESTIONS dataset. Furthermore, TAPEX yields strong results even with a small pre-training corpus, demonstrating its high efficiency. Finally, TAPEX achieves new state-of-the-art results on all experimental benchmarks, outperforming previous approaches by a large margin, including complicated table pre-training approaches with several heuristics in data processing. We will make our code, model, and data publicly available to facilitate future research.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Before diving into the details of our proposed table pre-training, we start by describing how to tackle downstream task fine-tuning with the encoder-decoder sequence generation paradigm. In this section, we first present the background of two fundamental table related downstream tasks: tablebased question answering (TableQA) and table-based fact verification (TableFV). Then we elaborate on our generative fine-tuning method in detail.  Figure 2: The illustration of the fine-tuning procedure in our method. During fine-tuning, we feed the concatenation of an NL sentence and its corresponding table taken from the downstream task to the model, and train it to output the answer (e.g., "Marisela Moreno Montero").
TAPEX: Table Pre-training via Learning a Neural SQL Executor
As mentioned in § 1, downstream tasks always involve joint reasoning over free-form NL sentences and tables. Therefore, examples of downstream tasks generally contain an NL sentence x and a (semi-)structured table T as the model input. Each NL sentence consists of K tokens as
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Each cell s i,j includes a list of tokens and corresponds to a table header c j . As for the output, there are variations among different tasks. In this paper, we focus on TableQA and TableFV. TableQA aims to retrieve table content to answer the user's question, and thus its output is either a list of cell values or number(s) calculated over the selected table region by aggregation functions (e.g., SUM). It is worth noting that for semi-structured tables, the answer may not be exactly table cell values, but their normalized forms (e.g., from 2k to 2,000), which makes downstream tasks more challenging (Oguz et al., 2020). As for TableFV, the output is a binary decision entailed or refused, indicating whether the NL sentence follows the fact indicated by the table.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
In this section, we present a generative approach for downstream task fine-tuning. Unlike previous works, we model both TableQA and TableFV as sequence generation tasks and leverage generative LMs to generate the output autoregressively. Taking TableQA as an example, given an NL question, our method generates the answer by decoding it in a word-by-word fashion.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Architecture Our method theoretically applies for any LM as long as it can generate sequence, such as GPT3 (Brown et al., 2020) and UniLM (Bao et al., 2020). In our experiments, we implemented our method based on BART (Lewis et al., 2020), a widely used pre-trained encoder-decoder model. BART follows a standard sequence-to-sequence Transformer architecture (Vaswani et al., 2017), with modifying ReLU activation functions to GeLU. It is pre-trained via corrupting sentences (i.e., randomly sampling length-variable spans and masking each one with a single [MASK] token) and then optimizing a reconstruction loss. As for the number of layers, we employ the BART Large configuration in our experiments, i.e., 12 layers are used in both the encoder and the decoder.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Model Input As illustrated in Figure 2, the input contains an NL sentence and its corresponding table. Encoding the NL sentence is relatively straightforward, while encoding the table is non-trivial since it exhibits underlying structures. In practice, we flatten the table into a sequence so that it can be fed directly into the model. By inserting several special tokens to indicate the table boundaries, a flattened table can be represented as T * = [HEAD], c 1 , · · ·, c N , [ROW], 1, r 1 , [ROW], 2, r 2 , · · ·, r M . Here [HEAD] and [ROW] are special tokens indicating the region of table headers and rows respectively, and the number after [ROW] is used to indicate the row index. Notably, we also separate headers or cells in different columns using a vertical bar | . Finally, we prefix the flattened table T * with the NL sentence x and feed them into the model encoder.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Model Output With attending on the encoder, the decoder is responsible for modeling the outputs of both TableQA and TableFV. For TableQA, the output is the concatenation of the answer(s) separated by commas, and the decoder generates it autoregressively. In this way, our model can readily support (almost) all operators and their compositions in TableQA. For TableFV, as BART does for sequence classification tasks (Lewis et al., 2020), the same input is fed into both the encoder and decoder, and a binary classifier upon the hidden state of the last token in the decoder is used for the output. Notably, our method can be easily extended to other table related tasks in a similar way. Figure 3: The illustration of the pre-training procedure in our method. During pre-training, we feed the concatenation of a sampled SQL query and a sampled table to the model, and train it to output the corresponding execution result (e.g., "Pairs").
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Fine-Tuning Strategy Since our approach can perform various downstream tasks on the same architecture, it can easily perform multi-task learning. Therefore, we explore two ways of finetuning, one for vanilla fine-tuning and the other for multi-task fine-tuning. The former is to fine-tune the model on each individual downstream task. The latter is inspired by TAPAS (Herzig et al., 2020) and T5 (Raffel et al., 2020), which first fine-tunes the model on related or similar intermediate downstream tasks and then continues to fine-tune it on the target downstream task.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Discussion Our approach comes with several advantages: (i) Flexibility: due to the powerful expressiveness of encoder-decoder models, our approach can readily adapt to (almost) any kind of output. (ii) Conveniency: our approach does not require any modification (e.g., table-specific masking) on pre-trained LMs, and can be trained in an end-to-end manner. (iii) Transferability: since we formulate downstream tasks as sequence generation tasks, which allows different tasks to share the same training protocol, it is easy to perform multi-task fine-tuning for our approach.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
As mentioned in § 1, TAPEX achieves efficient table pre-training by training LMs to mimic the behavior of a SQL execution engine. In this section, we illustrate how to conduct table pre-training from two aspects: the pre-training task and the pre-training corpus.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Following the MLM task in NL pre-training, existing works usually use reconstruction tasks for table pre-training. They generally take corrupted tables and NL sentences as input and try to recover the corrupted parts, in order to strengthen the linking between NL sentences and tables. While these pre-training tasks perform well, they tend to be less efficient since they usually require an extremely large pre-training corpus.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
To design efficient tasks for table pre-training, we argue that the key lies in the executability of tables. That is to say, structured tables enable us to perform discrete operations on them via programming languages such as SQL queries, while unstructured text does not. Taking this into account, TAPEX adopts SQL execution as the only pre-training task. As illustrated in Figure 3, the pre-training of TAPEX is similar to the procedure of the above generative fine-tuning. Given an executable SQL query and a table T , TAPEX first concatenates the SQL query and the flattened table T * to feed into the model encoder. Then it obtains the query's execution result through an off-the-shelf SQL executor (e.g., MySQL) to serve as the supervision for the model decoder. Intuitively, the pretraining procedure is to encourage a language model to be a neural SQL executor. We believe that if a language model can be trained to faithfully "execute" SQL queries and produce correct results, then it should have a deep understanding of tables.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Synthesizing the pre-training corpus is very important for table pre-training. Generally, there are two key factors: the table source and the SQL query sampling strategy.    (Yin et al., 2020), TAPEX works well even with only a few thousand tables. Therefore, instead of fetching noisy tables from the Web and then heuristically filtering them, we pick high-quality tables right from existing public datasets. Concretely, we randomly select nearly 1, 500 tables from the training set of WIKITABLEQUESTIONS (Pasupat & Liang, 2015) as the table source for our pre-training corpus. Notice that there is no overlap between the tables used in our pre-training and the tables used in the dev and test sets of all downstream tasks, so there is no data leakage problem.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Query Sampling Regarding the sampling of diverse SQL queries, there are various choices in the literature. We can either sample SQL queries according to a probabilistic context-free grammar , or instantiate SQL templates over different tables (Zhong et al., 2020a). In our experiments, we follow the latter, where SQL templates are automatically extracted from the SQUALL dataset (Shi et al., 2020b). An example SQL template is: SELECT num1 WHERE text1 = val1, where num1 and text1 correspond to a numeric column and a text column respectively, and val1 refers to one of the cell values with respect to the column text1. Given a SQL template, at each instantiation, we uniformly sample headers and cell values from a sampled table to fill the template, forming a concrete SQL query. Notably, SQL queries that execute with empty results are discarded, because empty results do not reflect much information about the executability of tables. This way, we can obtain a large-scale pre-training corpus with high quality.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
In this section, we evaluate TAPEX on different downstream tasks to verify its effectiveness.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
We evaluate the performance of our approach on weakly-supervised WikiSQL (WIKISQL-WEAK) (Zhong et al., 2017), WIKITABLEQUESTIONS (Pasupat & Liang, 2015), SQA (Iyyer et al., 2017), and TABFACT . Compared to WIKISQL-WEAK, which only requires filtering and optionally aggregating on table cell values, WIKITABLE-QUESTIONS requires more complicated reasoning capabilities. SQA is a conversational benchmark, which requires our approach to model the conversational context. Datset details can be found in Appendix A. For TableQA datasets, the evaluation metric is denotation accuracy, which checks whether the predicted answer(s) is equal to the ground-truth answer(s). It is worth noting that we evaluate our approach on WIKISQL-WEAK with answer annotations provided by TAPAS (Herzig et al., 2020), since nearly 2% of answers obtained from the official evaluation script are incorrect. For TABFACT, the evaluation metric is accuracy, which is calculated using the percentage of correct prediction.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
We implement our approach based on fairseq (Ott et al., 2019). During pre-training, we synthesize up to 5 million pairs of SQL queries and their execution results for   TAPEX. In the following, unless specified explicitly, all the experimental results are by default evaluated under the 5 million setting. Our pre-training procedure runs up to 50, 000 steps with a batch size of 256. It takes about 36 hours on 8 Tesla V100 GPUs to finish the pre-training. The best pre-training checkpoint is selected based on the loss on the validation set. For all downstream datasets, the fine-tuning procedure runs up to 20, 000 steps with a batch size of 128. For both pretraining and fine-tuning, the learning rate is 3×10 −5 .  Table 3 and Table 4 summarize the experimental results of various models on WIKISQL-WEAK, WIKITABLEQUESTIONS, SQA and TABFACT respectively. For both dev and test sets of all datasets, we report the median performance of our approach for five random runs.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
WIKISQL-WEAK As shown in Table 1, TAPEX outperforms all the baselines by a large margin.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
On the test set of WIKISQL-WEAK, TAPEX registers a denotation accuracy of 89.5%, which is 3.7% higher than BART and 2.3% higher than the previous best performance. This is significant since the previous best model has already utilized the execution-guided decoding. In short, TAPEX achieves a new state-of-the-art result on the well-known benchmark WIKISQL-WEAK.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
WIKITABLEQUESTIONS On the more challenging WIKITABLEQUESTIONS, TAPEX also achieves a new state-of-the-art denotation accuracy of 57.5%, surpassing the previous best system by 4.8% (Table 2). Meanwhile, we find that BART alone can only reach the denotation accuracy of 38.0%, much worse than the performance of previous pre-training models. We conjecture that the performance degradation could be attributed to the relatively small amount of training data in WIK-ITABLEQUESTIONS, which makes the adaptation of BART to tabular structures more challenging.  Figure 4: The visualization results of attention weights from other tokens to the cell "adrian lewis". Intuitively, the darker the color, the more closely the word is associated with "adrian lewis".
TAPEX: Table Pre-training via Learning a Neural SQL Executor
However, TAPEX delivers a dramatic improvement of 19.5% over BART, indicating that in the low data regime, the improvements introduced by TAPEX are often more significant.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
SQA Table 3 presents the performance of various models on the test set of SQA, where TAPEX again obtains a new state-of-the-art denotation accuracy in terms of both the conversation level (48.4%) and the sentence level (74.5%). This improvement is also a surprise to us since SQA is a conversational dataset while our pre-training task is context-free. Meanwhile, the substantial improvements of TAPEX over BART on SQA continues to verify the same observation that TAPEX alleviates the low resource issue.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
TABFACT Beyond TableQA, TAPEX also excels at TableFV. As shown in Table 4, TAPEX achieves new state-of-the-art results on all subsets of TABFACT. For example, it surpasses the previous best system by 4.0% on Test complex . The result shows that TAPEX endows BART with generic table understanding capabilities, which could be adapted to different downstream tasks, regardless of whether these tasks are highly similar to the TAPEX pre-training task or not.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Overall Results Experimental results on four datasets show that TAPEX can broadly improve the model ability on understanding tables, especially in the low data regime.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
As discussed in § 2.2, our approach can easily perform multi-task learning, thereby conferring benefits to downstream tasks. To verify it, we conducted multi-task fine-tuning experiments and obtained the following findings: (1) when initialized by BART, multi-task fine-tuning boosts the performance of the target task significantly;
TAPEX: Table Pre-training via Learning a Neural SQL Executor
(2) when initialized by TAPEX, the gain of multi-task fine-tuning tends to be marginal, suggesting that most of the "skills" (loosely speaking) gained by multi-task learning can be acquired by our table pre-training. Detailed results can be found in Appendix B.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
In this section, we carefully analyze our approach in terms of various aspects. Besides, we perform an exploratory analysis to provide more insights for future work, which can be found in Appendix C.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
SQL Execution by Pre-training In order to understand how well TAPEX performs SQL execution after pre-training, we analyze its performance on nearly 20, 000 held-out SQL queries over unseen tables. Overall, the SQL execution accuracy is relatively high, as TAPEX correctly "executes" 89.6% of the SQL queries 1 . In particular, TAPEX performs better on Filter, Aggregate and Superlative operators, indicating that it is highly accurate in table cell selection and table aggregating. Regarding Arithmetic and Comparative operators, TAPEX also does a good job, demonstrating its numerical reasoning skill on tables. To summarize, TAPEX has learned to be a neural SQL executor with good selection, aggregating and numerical capabilities.  Figure 4, TAPEX seems to focus more on the row and the header where a cell corresponds to. Taking the example from Figure 4, the attention weights imply that "adrian lewis" is closely associated with the first column "player" and the entire third row, which are the positions of "adrian lewis" in the structured table.     Table 5. One can find that TAPEX significantly boosts the performance on all operators, implying that it does enhance BART's capabilities for joint reasoning over text and tables.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
The Scale of Pre-training Corpus Figure 5 illustrates downstream performance with different scales of the pre-training corpus. It can be seen that even if our pre-training corpus is synthetic, scaling up the pre-training corpus generally brings positive effects. The observation is analogous to the one in language modeling (Brown et al., 2020): the larger the pre-training corpus, the better the downstream performance. By the comparison across different datasets, we can find that for simple tasks like WIKISQL-WEAK, the gains by scaling up pre-training corpus become marginal, while they remain non-trivial for complex tasks like TABFACT. Meanwhile, both downstream datasets in the low data regime show a positive trend by increasing the pre-training corpus. Conclusively, the scale matters when the downstream task is difficult, or the downstream dataset is relatively small.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
The Efficiency of Pre-training As mentioned in § 1, the pre-training efficiency of existing table pre-training approaches is relatively low, as they usually require an extremely large corpus. Therefore, taking WIKITABLEQUESTIONS as an example, we compare the pre-training efficiency of TAPEX with TAPAS (Herzig et al., 2020), TABERT (Yin et al., 2020) and GRAPPA (Yu et al., 2021a). It is worth noting that part of the pre-training corpus for GRAPPA comes from humanannotated, high-quality parallel data. As shown in Figure 6, TAPEX can yield very promising performance when using a much smaller pre-training corpus, indicating that our proposed SQL execution pre-training task is more efficient than other table pre-training tasks.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Limitations The first limitation of our approach is that it cannot ideally handle large tables. As mentioned above, we employ the table flattening technique to represent a table. It works well when the table is relatively small, but it becomes infeasible when the table is too large to fit in memory. In practice, we can compress tables by removing some unrelated rows or columns, which would decrease downstream performance. The second limitation is that the task of text-to-SQL cannot benefit from our proposed table pre-training. We have tried to apply TAPEX for a text-to-SQL task, where the input remains the same and the output converts to SQL. However, TAPEX does not show a significant advantage over BART. We attribute this to two factors: first, our synthetic pre-training corpus does not contribute to grounding, one of the most important factors for semantic parsing ; second, table reasoning capabilities (e.g., aggregate) learned by TAPEX may not be necessary for SQL generation. For example, a model could still understand an NL phrase "total" as the aggregation function "sum", even though it is unaware of the mathematical meaning of "sum". Table Pre-training The work most related to ours is table pre-training whose key factors include the pre-training corpus and the pre-training task. As for the pre-training corpus, most of previous works almost collect NL-table data to perform table pre-training. They either mined a large corpus of tables and their NL sentence contexts (Yin et al., 2020;Herzig et al., 2020), leveraged human-annotated parallel NL-table datasets for pre-training (Deng et al., 2021;Yu et al., 2021a), or synthesized a NL-table corpus using human-written templates (Yu et al., 2021a;Eisenschlos et al., 2020). Our work is different from theirs because we are the first to use pure synthetic SQL-table data for table pre-training, which allows us to automatically synthesize a diverse, large-scale, and high-quality pre-training corpus. As for the pre-training task, existing works proposed several pretraining tasks, such as Mask Column Prediction (Yin et al., 2020), Multi-choice Cloze at the Cell Level (Wang et al., 2021b) and Structure Grounding (Deng et al., 2021). Different from all of them, we present a novel SQL execution task to perform table pre-training.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Joint Understanding on Table and Text As our experiments are mainly on TableQA and TableFV, our work is also closely related to previous methods for these tasks. For TableQA, previous works almost formulate it as a weakly semantic parsing task (Liang et al., 2018;Wang et al., 2019a;, which always employ reinforcement learning to optimize semantic parsers over tables. Although these parsers produce logic forms (e.g., SQL), they have difficulties in training due to the large search space and the presence of spurious programs (Goldman et al., 2018). In addition, another promising line of work has emerged in recent advances (Mueller et al., 2019;Herzig et al., 2020), which aims at answering NL sentences without logical forms. This line of work predicts answer(s) by selecting cell values and optionally applying an aggregation operator to them. They can be easily trained, but their modeling ability is limited. For example, it is hard to support compound aggregation operators such as max(Year) -min(Year). What makes our approach different from these works is that we employ generative models to handle TableQA and can enjoy the end-toend training and flexibility simultaneously. For TableFV, previous works usually employ specialized architectures with limited scalability (Shi et al., 2020a;. For example, Zhong et al. (2020b) leveraged a graph construction mechanism, a semantic parser, and a semantic composition model to capture the connections among the NL sentence and the table. While the approach works well for TableFV, it is not easily applied to other table-related tasks. Compared with them, our approach works well for a variety of downstream tasks in the same architecture.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
In this paper, we present TAPEX, an execution-centric table pre-training approach whose corpus is automatically synthesized via sampling SQL queries and their execution results. TAPEX addresses the data scarcity challenge in table pre-training by learning a neural SQL executor on a diverse, large-scale, and high-quality synthetic corpus. Experimental results on four downstream datasets demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. Our work opens the way to exploit structured data by pre-training on synthetic executable programs, which is conceptually simple and has great potential to be extended to other research areas (e.g., knowledge base).
TAPEX: Table Pre-training via Learning a Neural SQL Executor
We would like to thank all the anonymous reviewers for their constructive feedback. The first author Qian is supported by the Academic Excellence Foundation of Beihang University for PhD Students.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
In this work, we present a novel pre-training approach for tabular data, which approximates the structural reasoning process of formal languages over tables to achieve efficient table pre-training. Different from previous works which employ web crawling to construct a large-scale NL-table corpus for pre-training, our pre-training corpus is synthesized via sampling SQL queries and their execution results on public tables. Compared with previous works, our pre-training corpus is more controllable with high-quality. For example, compared with TABERT which crawls 26 million noisy tables from the Web, our approach adopts 1, 500 high-quality tables from public datasets, which greatly alleviates the potential privacy and bias issues raised by web crawling. We evaluate our approach on two fundamental table-related tasks: table-based question answering and table-based fact verification. The former enables non-expert users to query databases without learning programming languages, while the latter helps users to verify whether a textual hypothesis is valid based on given tabular evidence. Experimental results on four well-known benchmark datasets show that our approach achieves new state-of-the-art results on all of them, especially in the low data regime.   
TAPEX: Table Pre-training via Learning a Neural SQL Executor
The dataset statistics are shown in Table 6, while Table 7 show example inputs and outputs for our model. Note that SQA is a conversation benchmark, and we directly concatenate the history and i-th question as the "sentence" part (x) in the input, as done in . 
TAPEX: Table Pre-training via Learning a Neural SQL Executor
In this section, we perform an exploratory analysis to provide more insights for future work. Concretely, we explore two interesting research questions: (1) How does the difficulty of SQL queries     Figure 8: The fine-grained performance of different SQL difficulty levels in pre-training on different question difficulty levels from WIKITABLEQUESTIONS dev set.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Fine-Grained Analysis To understand the impact from a fine-grained perspective, we divide questions from the WIKITABLEQUESTIONS dev set into the same four levels of difficulty, with the help of SQL query annotation for WIKITABLEQUESTIONS questions provided by SQUALL. All finegrained experimental results are presented in Figure 8. We can see that with the addition of harder SQL queries, the performance on questions at the same difficulty level are greatly improved. For example, the addition of Medium level SQL queries boosts the performance of Medium-level questions from 38.2% (≤ Easy) to 56.2% (≤ Medium), which is in line with expectations. More encouragingly, adding simpler SQL queries can even improve performance on harder questions. For example, compared to BART, the ≤ Medium pre-training leads to an impressive improvement of up to 13.1% in the performance of Hard-level questions.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Natural Language Generation Intuitively, compared to SQL queries, using NL sentences in pretraining is better for downstream tasks since the pre-training objective is nearly the same as the fine-tuning objective. However, it is non-trivial to obtain a fluent NL sentence which faithfully reflects the semantics of a SQL query. In this experiment, we follow Zhong et al. (2020a) to train a SQL-to-NL model and employ the model to translate SQL queries from the pre-training corpus into NL sentences. Concretely, our SQL-to-NL model is based on BART-Large (Lewis et al., 2020) and trained on the SQUALL dataset (Shi et al., 2020b), which contains nearly 9, 000 SQL-NL pairs. Then we apply the well-trained SQL-to-NL model to the pre-training corpus of TAPEX (0.5 Million) and obtain a NL pre-training corpus of the same size. By manually analyzing 100 sampled translated NL sentences, we are surprised to find that all NL sentences are fluent, and nearly 68% of them SELECT COUNT (*) WHERE Result = "won" AND Year > 1987 How many times did they win after 1987?
TAPEX: Table Pre-training via Learning a Neural SQL Executor
SELECT MAX (Chart Position) -MIN (Chart Position) WHERE Release date = "july 21, 1995"
TAPEX: Table Pre-training via Learning a Neural SQL Executor
What is the difference between the chart position of july 21, 1995 and the chart position of july 22, 1995?
TAPEX: Table Pre-training via Learning a Neural SQL Executor
SELECT Nation WHERE Nation != "Japan" AND Gold = (SELECT Gold WHERE Nation = "Japan" )
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Which other countries had the same number of gold medals as Japan?
TAPEX: Table Pre-training via Learning a Neural SQL Executor
SELECT Incumbent Electoral History GROUP BY Incumbent Electoral History ORDER BY COUNT ( * ) DESC LIMIT 1
TAPEX: Table Pre-training via Learning a Neural SQL Executor
Who has held the office the most?  are faithful to the semantics of the corresponding SQL queries. Table 10 presents some sampled SQL queries and their corresponding translated NL sentences. After obtaining the NL pre-training corpus, we follow the same pre-training and fine-tuning procedures as TAPEX to leverage it.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
We compare the performance of all downstream tasks between TAPEX with. SQL and TAPEX with. NL in Table 11. Surprisingly, the performance of TAPEX with. NL is comparable or even worse than the one of TAPEX with. SQL. For example, compared to using SQL queries in pre-training, using NL sentences causes a drop of 1.4% on WIKITABLE-QUESTIONS. We attribute such drop to the fact that the translated NL sentences contain some noise. Taking the second row in Table 11 as an example, the translated NL sentence includes extra information such as "in the 1989 major league baseball draft", which may interfere with the pre-training.
TAPEX: Table Pre-training via Learning a Neural SQL Executor
D FINE-GRAINED ANALYSIS OF SQL EXECUTION Figure 9 provides a fine-grained analysis of the SQL execution accuracies for each operator type.  Figure 9: The fine-grained statistics of typical operators, example SQLs, operator percentage and their execution accuracies on the held-out 20, 000 SQL queries.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Inspired by the massive success of pre-trained language models (LM) on free-form natural language (NL) tasks (Devlin et al., 2019;Dong et al., 2019;Raffel et al., 2020;Lewis et al., 2020), researchers have attempted to extend the pre-training to table data. Tables are a valuable form of data that organize information in a structured way. They often contain data that is organized in a more accessible manner than in unstructured texts. To adapt the pre-training paradigm on structured tab-  Deng et al., 2020;Zayats et al., 2021), adding auxiliary structure indicative embeddings (Herzig et al., 2020;, and designing table-specific pre-training objectives Yu et al., 2021a;Liu et al., 2022b,a). While these  methods are effective in understanding table structures, they increase the modeling complexity and  lack interpretability on why models learns table  reasoning skills during pre-training.  This paper presents a new table pre-training approach, named REASTAP, which enables a model  to efficiently learn table structure understanding  and table reasoning skills during pre-training. We first defined 7 table reasoning skills, such as numerical operation and temporal comparison. As shown in Figure 1, for each reasoning skill, a corresponding example generator was applied to synthesize Question Answering (QA) examples over tables. We modeled the pre-training task as a sequence generation task and pre-trained a sequence-to-sequence (seq2seq) LM to generate the answer to the synthetic questions. REASTAP is theoretically applicable to any seq2seq LM without a table-specific architecture design. Our key insight is that if a language model can be pre-trained to generate the answers to synthetic questions, which require various table reasoning skills, it should have a great  table structure understanding and table reasoning  Each template contains typed variables that are instantiated with content from the extracted table. Specifically, column col and cell value val are indexed to specify that val:i must be instantiated by a cell value from the i-th column. Some templates also regulate that the selected column and cell value must be date or number type. OPERATOR and ORDINAL correspond to operators and ordinal numerals that are instantiated according to the specific reasoning skill. And CONDITION:i can be 1) a cell value from the i-th column; or 2) a number/temporal comparison statement if the i-th column is date or number type. For example, the question from Figure 1 "Which Company Name, with Headquarter was United States, has the 4th Profit?" are generated from one of the "Numerical Comparison" templates: "Which col:1, with col:2 was CONDITION:2, has  Table 1: 7 reasoning skills with example for pre-training REASTAP. Variable names indicate permissible instantiations. col denotes a column name, val denotes a cell value, and indices denote that a cell value must originate from the specified column. OPERATOR and ORDINAL correspond to operators and ordinal numeral that are instantiated according to the specific reasoning skill, e.g., for 'Temporal Comparison', ORDINAL is replaced with a reasonable ordinal numeral such as "4th". And CONDITION:i can be 1) a cell value from the i-th column, or 2) number/temporal comparison statement (e.g. "later than 1967") if the i-th column is of number or date type. the ORDINAL col:3?" Once all variables in the sampled template were instantiated, we obtained question q. Then the example generator would programmatically return the corresponding answer a.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
After generating a vast number of QA examples for each reasoning skill, we had to sample pre-training data from these synthetic examples. In our setting, the portion of pre-training examples (Table 1) corresponding to each reasoning skill roughly matches the portion of logical operations defined in Tab-Fact (Chen et al., 2020b). We raised the portion of numerical operation skill as numerical reasoning is more challenging for models to learn. To increase the diversity of pre-training corpus, for each reasoning skill, we also sampled {SQL query, execution result} pairs from TAPEX (Liu et al., 2022b) pretraining corpus as complementary QA examples. The sampled pairs were categorised according to their function (e.g., COUNT, SUM). As a result, we obtained a total of 4M pairs of reasoning examples as the pre-training corpus for REASTAP.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
3 Pre-training REASTAP Task Formulation Each example in the synthetic pre-training corpus contains a question q and a semi-structured table T as the model input. The task objective is to generate an accurate answer string a = (a 1 , a 2 , . . . , a n ) given the question q and input table T :
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
where θ denotes the parameters of a seq2seq LM.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Model Architecture Our method is theoretically applicable to any seq2seq LM, such as T5 (Raf-fel et al., 2020) and GPT3 (Brown et al., 2020).
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
In our experiments, we implemented REASTAP based on BART (Lewis et al., 2020), a widely used Transformer-based pre-trained model (Vaswani et al., 2017) that has proved its effectiveness on various comprehension and text generation tasks.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
In our experiments, we chose BART-Large as a backbone, which has around 400M parameters and 12 layers in both encoder and decoder.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Data Serialization As illustrated in Figure 1, the input contains a question and its corresponding table. We flattened the table so that it can be fed directly into the encore-decoder model. Specifically, by inserting several special tokens to indicate the table boundaries, a flattened table is denoted as 
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
We evaluated REASTAP on three different types of downstream tasks to verify its effectiveness. The statistics and examples for each task are shown in Table 2, and Table 10 in the Appendix, respectively. The fine-tuning of REASTAP is similar to the procedure for pre-training discussed in section 3. We modeled both downstream tasks as sequence generation tasks and leverage generative LMs to generate the output autoregressively.  (Zhong et al., 2017) and WIKITQ (Pasupat and Liang, 2015) were used to evaluate REASTAP performance on Table QA tasks. WIKISQL-WEAK (Zhong et al., 2017) requires the models to perform filtering and optional aggregation on table cell values to answer the given a question. WIKITQ (Pasupat and Liang, 2015) requires a broader set of reasoning skills, thus is more challenging. The Table QA task formulation is the same as the REASTAP pretraining task. We used the denotation accuracy, which checks whether the predicted answers are equal to the ground truths, as evaluation metric. 
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
We used the accuracy (i.e., percentage of correct predictions) as evaluation metric.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Faithful Table-to-Text Generation We chose LOGICNLG (Chen et al., 2020a) to evaluate REASTAP performance on the Faithful Table-to-Text Generation task. Compared with previous Table-to-Text generation benchmarks (Wiseman et al., 2017;Balakrishnan et al., 2019;Parikh et al., 2020;Nan et al., 2022b), which primarily focus on surface-level realizations without much logical inference, LOGICNLG is tasked to generate statements that are logically entailed by the selected table region. Given the serialized input table with its selected columns as T , the objective is to generate a sentence y = (y 1 , y 2 , . . . , y n ) that is both fluent and factually correct:
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
To evaluate the logical fidelity of generated sentences, Chen et al. (2020a) proposed two modelbased evaluation methods: Parsing-based Evaluation (SP-Acc), and NLI-based Evaluation (NLI-Acc). SP-Acc directly extracts the meaning representation from the generated sentence and executes it against the    (Herzig et al., 2020) 49.9 50.4 TableFormer (Yang et al., 2022) 51.3 52.6 TaBERT  53.0 52.3 GraPPa (Yu et al., 2021a) 51.9 52.7 TAPEX (Liu et al., 2022b) 58.0 57.2 BART 37.1 37.5 REASTAP 58.3 58.6 noting that higher BLEU scores do not correlate with better logical fidelity (Nan et al., 2022a).
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
We implemented our models based on the fairseq library (Ott et al., 2019). We adopted BART-Large as the backbone model. For table pre-training, we synthesized and sampled 4M pairs of reasoning ex-amples. In the following sections, unless specified explicitly, all the experimental results were evaluated under the default settings of 4M reasoning examples and BART-Large configuration. Our pretraining procedure ran 80,000 steps with a batch size of 256, which took about 34 hours on an 8 NVIDIA A5000 24GB cluster. For downstream tasks, the fine-tuning procedure ran 30,000 steps with a batch size of 256. The best pre-training and fine-tuning checkpoints were both selected according to the validation loss.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Table QA On WIKISQL-WEAK, REASTAP outperforms all the baselines (Table 3). Specifically, on the test set of WIKISQL-WEAK, REASTAP achieves a denotation accuracy of 90.4%, which is 4.3% higher than BART and 1.2% higher than the previous best performance. On the more challenging WIKITQ, as shown in Table 4, REASTAP also surpasses the previous best system by 1.4%. It is worth noting that compared to WIKISQL-WEAK, WIKITQ contains much fewer tables and examples, which makes the adaptation of BART to tabular structures more challenging. Further REASTAP obtains an improvement of 21.1% over BART, indicating that in the low data regime, the improvements brought by REASTAP are more significant. We also evaluated REASTAP performance under low-resource settings (Section 6.1). Faithful Table-to-Text Generation Table 6 presents the results on LOGICNLG. It is observed that the BART backbone has already achieved competitive results in terms of both surface-level and logical-fidelity metrics. Compared with the BART backbone, REASTAP obtains slightly lower results on BLEU scores, which is reasonable since we con-  (Zhang et al., 2020) 73.3 73.2 85.5 67.2 -TaPas (Herzig et al., 2020) 81.0 81.0 92.3 75.6 83.9 TableFormer (Yang et al., 2022) 82.0 81.6 93.3 75.9 84.6 DecompTaPas (Yang and Zhu, 2021)    tinued pre-training REASTAP on our pre-training corpus that is irrelevant to the text generation task. However, REASTAP significantly improves the logical-fidelity scores, enhancing the SP-Acc and NLI-Acc by 4.7% and 5.5%, respectively. The results demonstrate that REASTAP can also help improve faithful text generation.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Experimental results on three different kinds of downstream tasks show that REASTAP can broadly improve BART's generic table reasoning capabilities, which could be adapted to different downstream tasks, regardless of whether the tasks are highly similar to the REASTAP pre-training task or not. In this section, we further analyze our approach in terms of various aspects to provide researchers with a deeper insight for future work.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
To further understand how well REASTAP learns table reasoning skills during pre-training, we con- ducted experiments under the low-resource setting, where we fine-tuned REASTAP on 20% and 5% of downstream task training data. As shown in Table 7, in the low-resource setting, the improve-   2) 58.6 (+0.3) WIKISQL-WEAK 47.6 (+10.5) 58.4 (+0.1) Table 9: Dev denotation accuracy of multi-task finetuning on the taget task WIKITQ. ments introduced by REASTAP are often more significant. For example, with only 5% training data of downstream tasks, REASTAP delivers a dramatic improvement of 20.0%, 21.4%, and 11.2% over BART on WIKISQL-WEAK, WIKITQ, and TABFACT, respectively. The results from the low-resource setting show that REASTAP endows BART with generic table reasoning capabilities. Figure 2 illustrates REASTAP performance on downstream tasks with different pre-training corpus scales. We found that increasing the pretraining corpus generally brings positive effects for all downstream tasks. Furthermore, for simple tasks like WIKISQL-WEAK, the gains by scaling up pre-training corpus are marginal, while for complex tasks like WIKITQ, it shows a positive trend by scaling up the pre-training corpus.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
We investigated the contributions of the 7 reasoning skills to the downstream task performance of REASTAP. We devised 8 variants of REASTAP: one was trained with examples from all reasoning skills, while others were trained with examples without one reasoning skill. For each reasoning skill, we sampled 150K examples from the pretraining corpus. We kept the scale of pre-training corpus the same (i.e., 900K). We chose WIKITQ for experiments, on which BART does not perform well. Results shown in Table 8 demonstrate that all reasoning skills can benefit the model performance on WIKITQ. Furthermore, we find that some reasoning skills, such as counting and temporal comparison, bring more improvements to the model compared to others.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
The analysis also helps us understand how the sets of pre-defined reasoning skills are injected during pre-training. When adopting REASTAP to a new downstream task that requires new reasoning skills different from existing seven reasoning skills, one can also inject the new reasoning skill into model during the pre-training in a similar way. Specifically, once the templates for the new reasoning skill are designed, the synthesis pipeline will generate new examples for pre-training. Pretraining REASTAP on these synthetic examples can help model learn the new reasoning skill.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
We further conducted multi-task fine-tuning experiments to explore whether REASTAP can benefit from the source task. We chose WIKISQL-WEAK and TABFACT for the source task, as their training datasets are relatively rich, and WIKITQ as the target task. Models were first fine-tuned on the source task and then fine-tuned on the target task. As shown in Table 9, multi-task fine-tuning delivers a significant improvement to the target task when initialized by BART; while the improvements are marginal when initialized by REASTAP. This is reasonable because most table reasoning skills acquired by multi-task learning have been injected into the model during the pre-training.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
In this paper, we propose REASTAP, a new table pre-training approach, which injects various pre-defined table reasoning skills into models via learning to generate correct answers of synthetic questions. Compared to previous work which design table-specific architectures, REASTAP is easy to implement and is theoretically applicable to any sequence-to-sequence LM. REASTAP is evaluated over four downstream benchmarks. The experimental results demonstrate that REASTAP achieves new state-of-the-art results on each of them. This includes the improvements on WIKISQL-WEAK denotation accuracy to 90.4% (+1.2%); WIKITQ denotation accuracy to 58.6% (+1.4%); TABFACT accuracy to 84.7% (+0.7%); and LOGICNLG SP-Acc to 54.8% (+4.0%), NLI-Acc to 89.2% (+3.6%). Further analysis demonstrates that REASTAP delivers a significant improvement to BART on the low-resource setting, indicating that our proposed pre-training approach can effectively improve the model's generic table reasoning capabilities.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
The main limitation of our approash is that we utilized a template-based method to synthesize pretraining corpus. Although such template-based approach ensures the faithfulness of generated QA examples and the diversity of reasoning process required to answer the questions, it limits the semantic diversity of questions. We believe future work could exploit 1) more different types of reasoning skills, such as advanced numerical reasoning skills required in the finance domain Chen et al., 2021b); 2) a more universal synthetic example generation pipeline; 3) extending models to tables with hierarchical structures (e.g., more than one row or column header) Zhao et al., 2022); 4) a more efficient training framework (Biesialska et al., 2020;Yoran et al., 2021) that can update models to learn newlydefined reasoning skills effectively.
ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples
Tables used in our synthetic pre-training corpus are collected and extracted from the 02-20-2022 Wikipedia dump 2 , which is publicly available under the Creative Commons Attribution-ShareAlike 3.0 License and the GNU Free Documentation License. The licenses permit us to compose, modify, publish, and distribute additional annotations upon the original content.