How has active learning been used for semantic parsing?
Active Learning. AL is to select the most valuable unlabeled instances to be annotated in order to maximize the model’s performance and hence reduce the annotation cost for data-hungry machine learning models. AL has been used to MT (Haffari and Sarkar, 2009), sequence labelling (Vu et al., 2019), text classification (McCallum et al., 1998; Vu et al., 2023), and semantic parsing (Duong et al., 2018; Ni et al., 2020; Li and Haffari, 2023). Following most deep learning AL methods (Duong et al., 2018; Ni et al., 2020; Li and Haffari, 2023), our work also adopts a pool-based query strategy, which means we sample batches from a large pool of unlabelled data instead of evaluating examples one by one from an incoming stream. Among all the AL for semantic parsing works, Li and Haffari (2023) is the one most similar to ours, which selects utterances to be translated. However, they do not utilize MT systems.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Multilingual semantic parsing aims to leverage the knowledge from the high-resource languages to improve low-resource semantic parsing, yet commonly suffers from the data imbalance problem. Prior works propose to utilize the translations by either humans or machines to alleviate such issues. However, human translations are expensive, while machine translations are cheap but prone to error and bias. In this work, we propose an active learning approach that exploits the strengths of both human and machine translations by iteratively adding small batches of human translations into the machine-translated training set. Besides, we propose novel aggregated acquisition criteria that help our active learning method select utterances to be manually translated. Our experiments demonstrate that an ideal utterance selection can significantly reduce the error and bias in the translated data, resulting in higher parser accuracies than the parsers merely trained on the machine-translated data.
[{"first":"Zhuang","last":"Li","middle":[]},{"first":"Lizhen","last":"Qu","middle":[]},{"first":"Philip","last":"Cohen","middle":[]},{"first":"Raj","last":"Tumuluri","middle":[]},{"first":"Gholamreza","last":"Haffari","middle":[]}]
Merging Weak and Active Supervision for Semantic Parsing
A semantic parser maps natural language commands (NLs) from the users to executable meaning representations (MRs), which are later executed in certain environment to obtain user-desired results. The fully-supervised training of such parser requires NL/MR pairs, annotated by domain experts, which makes them expensive to collect. However, weakly-supervised semantic parsers are learnt only from pairs of NL and expected execution results, leaving the MRs latent. While weak supervision is cheaper to acquire, learning from this input poses difficulties. It demands that parsers search a large space with a very weak learning signal and it is hard to avoid spurious MRs that achieve the correct answer in the wrong way. These factors lead to a performance gap between parsers trained in weakly- and fully-supervised setting. To bridge this gap, we examine the intersection between weak supervision and active learning, which allows the learner to actively select examples and query for manual annotations as extra supervision to improve the model trained under weak supervision. We study different active learning heuristics for selecting examples to query, and various forms of extra supervision for such queries. We evaluate the effectiveness of our method on two different datasets. Experiments on the WikiSQL show that by annotating only 1.8% of examples, we improve over a state-of-the-art weakly-supervised baseline by 6.4%, achieving an accuracy of 79.0%, which is only 1.3% away from the model trained with full supervision. Experiments on WikiTableQuestions with human annotators show that our method can improve the performance with only 100 active queries, especially for weakly-supervised parsers learnt from a cold start.
[{"first":"Ansong","last":"Ni","middle":[]},{"first":"Pengcheng","last":"Yin","middle":[]},{"first":"Graham","last":"Neubig","middle":[]}]
Active Learning for Multilingual Semantic Parser
Current multilingual semantic parsing (MSP) datasets are almost all collected by translating the utterances in the existing datasets from the resource-rich language to the target language. However, manual translation is costly. To reduce the translation effort, this paper proposes the first active learning procedure for MSP (AL-MSP). AL-MSP selects only a subset from the existing datasets to be translated. We also propose a novel selection method that prioritizes the examples diversifying the logical form structures with more lexical choices, and a novel hyperparameter tuning method that needs no extra annotation cost. Our experiments show that AL-MSP significantly reduces translation costs with ideal selection methods. Our selection method with proper hyperparameters yields better parsing performance than the other baselines on two multilingual datasets.
[{"first":"Zhuang","last":"Li","middle":[]},{"first":"Gholamreza","last":"Haffari","middle":[]}]
Active learning for deep semantic parsing
Semantic parsing requires training data that is expensive and slow to collect. We apply active learning to both traditional and “overnight” data collection approaches. We show that it is possible to obtain good training hyperparameters from seed data which is only a small fraction of the full dataset. We show that uncertainty sampling based on least confidence score is competitive in traditional data collection but not applicable for overnight collection. We propose several active learning strategies for overnight data collection and show that different example selection strategies per domain perform best.
[{"first":"Long","last":"Duong","middle":[]},{"first":"Hadi","last":"Afshar","middle":[]},{"first":"Dominique","last":"Estival","middle":[]},{"first":"Glen","last":"Pink","middle":[]},{"first":"Philip","last":"Cohen","middle":[]},{"first":"Mark","last":"Johnson","middle":[]}]
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Multilingual semantic parsing allows a single model to convert natural language utterances from multiple languages into logical forms (LFs). Due to its wide applications in various research areas, e.g. multilingual question answering and multilingual virtual assistant, multilingual semantic parsing has drawn more attention recently (Zou and Lu, 2018;Sherborne et al., 2020;Li et al., 2021a).
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Training a multilingual semantic parser (MSP) requires training data from all target languages. However, there is a severe imbalance of data availability among languages for current multilingual semantic parsing research. The utterances in most current semantic parsing datasets are in English, while non-English data is scarce.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
To overcome the data imbalance issue, prior studies translate utterances in the MSP datasets from high-resource languages (e.g. English) to the target low-resource languages of interest by either human translators (Susanto and Lu, 2017;Duong et al., 2017;Li et al., 2021a) or automatic machine translation (MT) (Moradshahi et al., 2020;Sherborne et al., 2020). Unfortunately, human translation (HT), though effective, is cost-intensive and timeconsuming. While the cost of MTs is much lower than that of HTs, the low quality of the machinetranslated utterances severely weakens the performance of the MSPs in the target languages.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
We observe that the quality of MTs is lower than that of HTs, mainly due to translation bias and errors. First, MT systems are likely to be influenced by algorithmic bias. Hence, the outputs of MT systems are generally less lexically and morphologically diverse than human translations (Vanmassenhove et al., 2021). So, there is a lexical distribution discrepancy between the machine-translated and the human-generated utterances. Second, MT systems are prone to generate translations with errors (Daems et al., 2017).
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Prior study (Moradshahi et al., 2020) demonstrates that adding only a small portion of humantranslated data into the complete set of machinetranslated training data significantly improves the MSP performance on the test set of the target language. Given this observation, we propose a novel annotation strategy based on active learning (AL) that benefits from both Human translations and Automatic machine Translations (HAT). It initially machine-translates all utterances in training sets from the high-resource languages to target languages. Then, for each iteration, HAT selects a subset of utterances from the original training set to be translated by human translators, followed by adding the HT data to the MT training data. The multilingual parser is trained on the combination of both types of translated data.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
We further investigate how HAT can select utterances whose HTs maximally benefit the parser performance. We assume the performance improve-ment is ascribed to the less biased and erroneous training set in a mixture of the MT and HT data. We have found that resolving the bias and error issues for the translations of the most semantically diversified and representative utterances improves the parser performance to the greatest extent. Given this assumption, we provide an Aggregated acquisition function that scores the utterances on how much their HTs can mitigate the Bias and Error issues for learning the multilingual parsers (ABE). It aggregates four individual acquisition functions, two of which measure the error and bias degree for the translations of the source utterances. The other two encourage the selection of the most representative and semantically diversified utterances.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Our key contributions are as follows:
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• We propose a novel AL procedure, HAT, that benefits from two popular annotation strategies for training the MSP. HAT greatly boosts the performance of the parser trained on MT data while it requires only a small extra human annotation cost. With only 16% of total utterances translated by humans, the parser accuracies on the multilingual GEOQUERY (Susanto and Lu, 2017) and NLMAP (Haas and Riezler, 2016) test sets can be improved by up to 28% and 5%, respectively, compared to the accuracies of those trained on machine-translated data, and are only up to 5% away from the ORACLE parsers trained on all human data.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• We propose an aggregated acquisition function, coined ABE, specifically designed to select utterances where their HTs mitigate translation bias and error for learning a good MSP. Compared to other SOTA acquisition baselines, given the same selection budget, our experiments consistently show ABE consistently results in the less biased and erroneous training sets and higher parser accuracies on the multilingual GEOQUERY and NLMAP test sets.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
An MSP is a parametric model P θ (y|x) that maps a natural language utterance x ∈ X into a formal meaning representation y ∈ Y, where X = l∈L X l includes utterances in different languages L. The standard training objective for a multilingual parser is,
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
where D L = l∈L D l includes training data where utterances are from multiple languages L.  Although using an MT system to train an MSP is cost-effective, the parser performance is usually much lower than the one trained with humantranslated data. For example, as shown in Table 1, the parsers trained on HTs all have significantly higher accuracies than those trained on MTs in different settings. Such performance gaps are due to two major issues of the MT data, discussed below.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Translation Bias. Many existing MT systems amplify biases observed in the training data ( Vanmassenhove et al., 2021), leading to two problems that degrade the parsers' performance trained on MT data:
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• The MTs lack lexical diversity (Vanmassenhove et al., 2021). As shown in Table 1, MTLD (Vanmassenhove et al., 2021) values show that the HTs of utterances in multilingual GEOQUERY and NLMAP are all more lexically diversified than MTs. Several studies (Shiri et al., 2022;Wang et al., 2015;Zhuo et al., 2023;Huang et al., 2021) indicate that lexical diversity of training data is essential to improving the generalization ability of the parsers.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• The lexical distribution of the biased MTs is different to the human-written text. The two metrics, Jensen-Shannon (JS) divergence (Manning and Schutze, 1999) and MAUVE (Pillutla et al., 2021), in Table 1 show the HTs of utterances in GEOQUERY and NLMAP are more lexically close to the human-generated test sets than MTs.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Translation Error. MT systems often generate translation errors due to multiple reasons, such as underperforming MT models or an absence of contextual understanding (Wu et al., 2023;, leading to discrepancies between the source text and its translated counterpart. One common error type is mistranslation (Vardaro et al., 2019), which alters the semantics of the source sentences after translation. Training an MSP on the mistranslated data would cause incorrect parsing output, as LFs are the semantic abstraction of the utterances. BT Discrepancy Rate in Table 1 demonstrates the mistranslation problem is more significant in the machine-translated datasets.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
To mitigate the negative effect of translation bias and error in the MT data, we propose HAT, which introduces extra human supervision to machine supervision when training the MSPs. Two major intuitions motivate our training approach:
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• Adding the HTs to the training data could enrich its lexical and morphological diversity and ensure that the lexical distribution of the training data is closer to the human test set, thus improving the parsers' generalization ability (Shiri et al., 2022;Wang et al., 2015).
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• HTs are less erroneous than MTs (Freitag et al., 2021). The parser could learn to predict correct abstractions with less erroneous training data.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Our HAT AL setting considers only the bilingual scenario. One of the languages is in high-resource, and the other one is in low-resource. However, it is easy to extend our method to more than two languages. We assume access to a well-trained black-box multilingual MT system, g mt (·), and a semantic parsing training set that includes utterances in a high-resource language l s (e.g. English) paired with LFs,
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
with utterances in high and low-resource languages, respectively. Each utterance x s in D s is translated into the utterancê x t = g mt s→t (x s ) in the target language l t by the MT system,
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
The goal of our AL method is to select an optimal set of utterances from the training data in the source language,D s ∈ D s , and ask human translators to translate them into the target language, denoted byD t = g ht s→t (D s ), for training a semantic parser on the union ofD t and D t . The selection criterion is based on the acquisi-tion functions that score the source utterances. Following the conventional batch AL setting (Duong et al., 2018), there are Q selection rounds. At the qth round, AL selects utterances with a budget size of K q .
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
The detailed HAT AL procedure iteratively performs the following steps as in Algorithm. 1.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Algorithm 1: HAT procedure Input :Initial training set D 0 = Ds ∪Dt, source utterance pool Ds, budget size Kq, number of selection rounds Q, human annotators g ht (·) Output :A well-trained multilingual parser P θ (y|x) # Train the initial parser on the initial data Update θ of P θ (y|x) with ∇ θ L(θ) on D 0 Evaluate P θ (y|x) on Ts and Tt Estimate the acquisition function ϕ(·)
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
# Translate the utterances inD q s into the target language lt by human annotators
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
# Add the human-translated data into the training data D q = Ds ∪Dt ∪D q t # Train the parser on the updated data Update θ of P θ (y|x) with ∇ θ L(θ) on D q Evaluate P θ (y|x) on Ts and Tt Re-estimate ϕ(·) end
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
The acquisition functions assign higher scores to those utterances whose HTs can boost the parser's performance more than the HTs of the other utterances. The prior AL works (Sener and Savarese, 2018;Zhdanov, 2019;Nguyen and Smeulders, 2004) suggest that the most representative and diversified examples in the training set improve the generalization ability of the machine learning models the most. Therefore, we provide a hypothesis that we should select the representative and diversified utterances in the training set, whose current translations have significant bias and errors. We postulate fixing problems of such utterances improves the parsers' performance the most. We derive four acquisition functions based on this hypothesis to score the utterances. Then, ABE aggregates these acquisition functions to gain their joint benefits. In each AL round, the utterances with the highest ABE scores are selected.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Translation Bias. We assume an empirical conditional distribution, P q e (x t |x s ), for each utterance x s in D s at qth AL selection round. Intuitively, the x s with the most biased translations should be the one with the most skewed empirical conditional distribution. Therefore, we measure the translation bias by calculating the entropy of the empirical conditional distribution, H(P q e (x t |x s )), and select the x s with the lowest entropy. Since the translation space X t is exponentially large, it is intractable to directly calculate the entropy. Following (Settles and Craven, 2008), we adopt two approximation strategies, N-best Sequence Entropy and Maximum Confidence Score, to approximate the entropy.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• N-best Sequence Entropy:
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
It is difficult to obtain the empirical distribution as we know neither of the two distributions that compose the empirical distribution. Therefore, we use distillation training (Hinton et al.) to train a translation model that estimates P q e (x t |x s ) on all the bilingual pairs (x s , x t ) in the MSP training data D q . Another challenge is that D q is too small to distil a good translation model that imitates the mixture distribution. Here, we apply a bayesian factorization trick that factorizes P q e (x t |x s ) = y∈Y P q e (x t |y)P q e (y|x s ), where y ranges over LFs representing the semanics. As there is a deterministic mapping between x s and the LF, P q e (y|x s ) is an one-hot distribution. Thus, we only need to estimate the entropy, H(P q e (x t |y)). This has a nice intuition: the less diversified data has less lexically diversified utterances per each LF. Note that if we use this factorization, all x s that share the same LF have the same scores.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
We use the lightweight, single-layer, recurrent neural network-based Seq2Seq model to estimate P q e (X t |x s ) or P q e (x t |y). It only takes approximately 30 seconds to train the model on GEO-QUERY. Ideally, every time a new source utterance x s is selected, P q e (x t |x s ) should be re-estimated. However, we only re-estimate P q e (x t |x s ) once at the beginning of each selection round to reduce the training cost.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Translation Error. Similar to Haffari et al.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
(2009), we leverage back-translations (BTs) to measure the translation error. We conjecture that if the translation quality for one source utterance x s is good enough, the semantic parser should be confident in the LF of the source utterance conditioned on its BTs. Therefore, we measure the translation error for each x s as the expected parser's negative log-likelihood in its corresponding LF y xs over all the BTs of x s , E P q e (xt|xs) [− log(P q θ (y xs |g mt t→s (x t )))], where P q θ is the parser trained at qth round. To approximate the expectation, we apply two similar strategies as mentioned in Translation Bias.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• N-best Sequence Expected Error:
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
where N yx s is the set of translations in D q that share the same LF y xs with x s . We only backtranslate utterances in D q to reduce the cost of BTs.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• Maximum Error:
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
We use the same distilled translation model P q e (x t |x s ) used in Translation Bias.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Density. The previous AL works ( Nguyen and Smeulders, 2004;Donmez et al., 2007) have found that the most representative examples improve the model performance the most. Therefore we desire to reduce the translation error and bias for the translations of the most representative source utterances. As such, the utterances should be selected from the dense regions in the semantic space,
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
We use kernel density estimation (Botev et al., 2010) with the exponential kernel to estimate P (x s ), while other density estimation methods could be also used. The feature representation of x s for density estimation is the average pooling of the contextual sequence representations from the MSP encoder. The density model is re-estimated at the beginning of each query selection round.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Semantic Diversity. The role of the semantic diversity function is twofold. First, it prevents the AL method from selecting similar utterances. Resolving the bias and errors of similar utterances in a small semantic region does not resolve the training issues for the overall dataset. Second, semantic diversity correlates with the lexical diversity, hence improving it also enriches lexical diversity. (9) where c(x s ) maps each utterance x s into a cluster id and S is the set of cluster ids of the selected utterances. We use a clustering algorithm to diversify the selected utterances as in (Ni et al., 2020;Nguyen and Smeulders, 2004). The source utterances are partitioned into |C| clusters. We select one utterance at most from each cluster. Notice the number of clusters should be greater than or equal to the total budget size until current selection round, |C| ≥ q i=1 K i . The clusters are re-estimated every round. To ensure the optimal exploration of semantic spaces across different query rounds, we adopt Incremental K-means (Liu et al., 2020) as the clustering algorithm. At each new round, Incremental K-means considers the selected utterances as the fixed cluster centres, and learns the new clusters conditioned on the fixed centres. The feature representation of x s for Incremental K-means is from MSP encoder as well.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Aggregated Acquisition. We aggregate the four acquisition functions into one,
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
where α k 's are the coefficients. Each ϕ k (x s ) is normalized using quantile normalization (Bolstad et al., 2003). Considering the approximation strategies we employ for both Translation Bias and Translation Error, ABE can be denoted as either ABE(N-BEST) or ABE(MAX). The term ABE(N-BEST) is used when we apply N-best Sequence Entropy and N-best Sequence Expected Error. On the other hand, ABE(MAX) is used when we implement Maximum Confidence Score and Maximum Error strategies.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Datasets. We evaluate our AL method for MSP on two datasets, GEOQUERY (Susanto and Lu, 2017) and NLMAP (Haas and Riezler, 2016) with multilingual human-translated versions. GEO-QUERY includes 600 utterances-LF pairs as the training set and 280 pairs as the test set. NLMAP includes 1500 training examples and 880 test examples.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
In our work, we consider English as the resourcerich source language and use Google Translate System 1 to translate all English utterances in GEO-QUERY into German (De), Thai (Th), Greek (El) and the ones in NLMAP into German, respectively. The AL methods actively sample English utterances, the HTs of which are obtained from the multilingual GEOQUERY and NLMAP.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Active Learning Setting. The HAT active learning procedure performs five rounds of query, which accumulatively samples 1%, 2%, 4%, 8% and 16% of total English utterances in GEOQUERY and NLMAP. We only perform five rounds as we found the performance of the multilingual parser is saturated after sampling 16% of examples with most acquisition functions. Baselines. We compare ABE with eight acquisition baselines and an oracle baseline.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
1. Random randomly selects English utterances in each round.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
2. Cluster (Ni et al., 2020;Li et al., 2021b) partitions the utterances into different groups using K-means and randomly selects one example from each group. 9. ORACLE trains the parser on the combination of English data, machine-translated data, and the complete set of human-translated data.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Evaluation. We evaluate the AL methods by measuring the accuracy of the MSP, the bias of the training set, and the semantic discrepancy rate between the selected utterances and their BTs.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• Accuracy: To evaluate the performance of the MSP, we report the accuracy of exactly matched LFs as in (Dong and Lapata, 2018) at each query round. As the parser accuracies on the English test sets are not relevant to evaluating the active learning method, we only report the accuracies on the test sets in the target languages. See Appendix A.2 for the English results.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• Bias of the Training Set: We use three metrics to measure the bias of the training data in the target language at each query round.  , 2005) reports the mean length of word strings in the utterances inD t ∪D q t that maintain a given TTR (Templin, 1957) value, where TTR is the ratio of different tokens to the total number of tokens in the training data. MTLD evaluate the lexical diversity of the training set.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
• BT Discrepancy Rate: Since we do not possess bilingual knowledge, we use BT to access the translation quality (Tyupa, 2011). At each query round, we randomly sample 100 utterances from the utterances selected by each acquisition in 5 seeds' experiments. The BT is obtained by using Google Translation to translate the MTs of the 100 sampled utterances back to English. Two annotators manually check the percentage of the BTs which are not semantically equivalent to their original utterances. We only consider a BT discrepancy when both annotators agree. Ideally, the utterances with fewer mistranslations would see fewer semantic discrepancies between the BTs and the original.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Effectiveness of HAT. Fig. 1  Bias, Error and Parser Performance. As in Table 2, we also measure the bias of the training set and the BT discrepancy rates of the selected utterances at the final selection round for each acquisition function. We can see that the parser accuracy directly correlates with the training set's bias degree. The bias of the training set acquired by RANDOM, TRAFFIC and CLUSTER, LFS-LC-D, and ABE score better in general than the other baselines in terms of the bias metrics, resulting in a better parser accuracy. RTTL and LCS(FW) that   select utterances with more erroneous translations do not necessarily guarantee better accuracies for parsers. Our following ablation study shows that the parser performance can be improved by correcting the translation errors for the most representative utterances. 
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Influence of different Acquisition Functions. As in Fig. 2, we evaluate the effectiveness of each acquisition by observing how removing each acquisition from ABE(N-BEST) influences the parser performance, the bias of the training set and the BT Discrepancy rate of the selected utterances. We can see that removing all terms degrades the parser performance. However, each acquisition contributes to the parser accuracy due to different reasons. Translation Bias and Semantic Diversity contribute to the parser performance mainly due to alleviating the bias of the training set. Excluding Translation Bias does not influence the lexical diversity, while the lexical similarity between the training and test sets becomes lower. Removing Semantic Diversity drops the lexical similarity as well. But it more seriously drops the lexical diversity when the sampling rates are high.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Removing Translation Error significantly decreases the parser accuracy and BT Discrepancy rate in the low sampling regions. However, when the selection rate increases, gaps in parser accuracies and BT Discrepancy rates close immediately. Translation Error also reduces the bias by introducing correct lexicons into the translations.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Removing Semantic Density also drops the parser performance as well. We inspect that Semantic Density contributes to parser accuracy mainly by combing with the Translation Error term. As in Appendix A.3, using Translation Error or Semantic Density independently results in inferior parser performance. We probe that Translation Error tends to select tail utterances from the sparse semantic region given the TSNE (Van der Maaten and Hinton, 2008) plots at Appendix A.7.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Influence of MT Systems. As in Fig. 3 (Right), at all query rounds, the multilingual parsers perform better with MT data in the training set, showing that MT data is essential for improving the parser's performance when a large number of HTs is not feasible. The quality of the MT data also significantly influences the parser performance when having no HT data in the training set. The accuracy difference between the parsers using Google and Bing translated data is greater than 10% when active learning has not been performed. However, after obtaining the HT data by HAT, the performance gaps close immediately, although the MT data of better quality brings slightly higher performance. When having all utterances translated by humans, the performance differences between parsers with different MT systems can be negligible. Fig. 3 also demonstrates that ABE(N-BEST) outperforms RANDOM, a strong acquisition baseline, with all three different MT systems. ABE(N-BEST) is also more robust to the MT systems than RAN-DOM. The performance gaps for the parsers with ABE(N-BEST) are much smaller than those with RANDOM when applying different MT systems. Figure 3: The parser accuracies across various query rounds on the GEOQUERY(DE) test set. We use two selection methods: ABE(N-BEST) (shown on the left) and RANDOM (shown on the right). For each method, we use data from different MT systems -Google, Bing, and our bespoke manually trained MT system. This manual MT system was developed without any pretraining weight, utilizing a limited set of bilingual data. The model architecture was based on the framework proposed by Ott et al. (2018). In addition to these, we also conducted tests without utilizing any MT data.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
We have tackled the problem of data imbalance when adapting an MSP to a low-resource language. We presented methods to efficiently collect a small amount of human-translated data to reduce bias and error in the training data, assuming a realistic scenario with an MT system and budget constraints for human annotation. Our experiments show that by manually translating only 16% of the dataset, the parser trained on this mixed data outperforms parsers trained solely on machine-translated data and performs similarly to the parser trained on a complete human-translated set.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
One of the limitations is the selection of hyperparameters. At present, we determine the optimal hyperparameters based on the performance of the selection methods on an existing bilingual dataset. For example, to identify the appropriate utterances to be translated from English to German, we would adjust the hyperparameters based on the performance of the methods on existing datasets in English and Thai. However, this approach may not always be feasible as such a dataset is not always available, and different languages possess distinct characteristics. As a result, the process of tuning hyperparameters on English-Thai datasets may not guarantee optimal performance on English-German datasets. As a future direction, we intend to investigate and develop more effective methods for hyperparameter tuning to address this limitation. 
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
A.1 Sensitivity Analysis Fig. 4 shows the parser results on GEOQUERY(DE) test sets when we apply different coefficients or cluster sizes to the four independent acquisitions in ABE(N-BEST). As we can see, tuning the parameters on an existing bilingual dataset does not necessarily bring optimal parser performance, indicating there is still potential in our approach if we can find suitable hyperparameter tuning methods. Another finding is that the ABE(N-BEST) is robust to the hyperparameters changes. Although changing the weights or cluster sizes for each term could influence the parser performances, the parser accuracies do not drop significantly. In addition, we have found that the Semantic Density and Semantic Diversity are more critical to ABE(N-BEST) as there are more fluctuations in the parser accuracies when we adjust the parameters of Semantic Density and Semantic Diversity. Figure 4: The parser accuracies across various query rounds on the GEOQUERY(DE) test set by employing the ABE(N-BEST) method. This method incorporates varying coefficients, denoted by α, for each term. In addition to this, we also analyze the influence of the proportional rate, denoted by k, which represents the number of clusters in proportion to the budget size at each round.   
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
As in Fig. 7, at several query rounds, the parser accuracy can be 3% -6% higher than that using no factorization in N-best Sequence Entropy. But factorization does not help ABE(MAX) improve the parser performance at all.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
As in Fig. 8, we combine Translation Error with different acquisition terms in ABE(N-BEST).
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Combing Translation Error and Translation Error achieves the best result in the low sampling regions. The accuracy is even 4% higher than the aggregated acquisition, ABE(N-BEST), when the sampling rate is 1%, suggesting that resolving translation error issues for semantically representative utterances benefits the parser more than resolving issues for tail utterances. 
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
We observe that the BT discrepancy patterns vary as in Fig. 9. For instance, the semantics of the BTs for Thai in GEOQUERY are altered dramatically due to the incorrect reorder of the words. Within NLMAP, the meanings of some German locations are inconsistent after the BT process.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Fig . 10 shows the T-SNE plot of the representations of sampled utterances among all the utterances in the training set using ABE(N-BEST) and its various ablation settings. We encode the utterances with the pre-trained language model in the encoder of BERT-LSTM. We can see if we only use Semantic Density alone, the utterances are more likely to be in the dense region while not semantically diversified. On the contrary, the Translation Error tends to select tail utterances in the sparse semantic regions while they also lack semantic diversity. Both terms independently result in inferior parser performances. The Translation Bias and Translation Diversity collect utterances from diverse areas, thus providing better parser accuracies as in Fig. A.3. Removing Semantic Diversity from ABE(N-BEST) drops the parser performance most. As we observe, after removing Semantic Diversity, the utterances become more semantically similar compared to the utterances selected by ABE(N-BEST). Overall, the T-SNE plot can be supplementary proof to our claim that we should retain the representativeness and diversity of the utterances to guarantee the parser performance. 
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
The maximum likelihood estimation objective of our parser is:
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
where x t is latent for most source utterance x s . We assume P e (x t |x s ) is a variational distribution.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
If we assume a conditional independence:
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
where E denotes the expectation function over a specified distribution and D KL denotes the Kullback-Leibler divergence between two distributions. Therefore the ELBO of log P θ (y, x s ) is E Pe(xt|xs) [log( P θ (xt,y,xs) Pe(xt|xs) )].
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
ELBO(P θ (y, x s )) = E Pe(xt|xs) [log( P θ (x t , y, x s ) P e (x t |x s ) )]
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
= E Pe(xt|xs) [log P θ (y|x t )] − D KL (P e ||P θ ) + log P θ (x s )
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
9526
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
A For every submission:
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
A1. Did you describe the limitations of your work?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
In the limitation section.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
A2. Did you discuss any potential risks of your work?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
No risks.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
A3. Do the abstract and introduction summarize the paper's main claims?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
In the abstract and conclusion sections.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
A4. Have you used AI writing assistants when working on this paper?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Grammarly and Quillbot. It helps me resolve writing issues and fix writing errors. Basically, I used them for all sections.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
B Did you use or create scientific artifacts?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Left blank.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
B1. Did you cite the creators of artifacts you used? No response.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
No response.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
In the Experiment section.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
The space is limited. I am just using 4 V100 GPUs for all my experiments.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
The space is limited.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
In the experiment section.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
D Did you use human annotators (e.g., crowdworkers) or research with human participants?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Space is limited.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
We just use local students.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
Left blank.
The Best of Both Worlds: Combining Human and Machine Translations for Multilingual Semantic Parsing with Active Learning
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.
Merging Weak and Active Supervision for Semantic Parsing
Semantic parsing maps a user-issued natural language (NL) utterance to a machine-executable meaning representation (MR), such as λ−calculus (Zettlemoyer and Collins 2005), SQL queries (Dong and Lapata 2018) or general purpose programming languages (e.g., Python) (Yin and Neubig 2018). These MRs can then be executed in a certain environment (e.g., a knowledge base, KB) to achieve the goal of users, such as querying a KB using natural language.
Merging Weak and Active Supervision for Semantic Parsing
Weakly-supervised semantic parsing aims to solve this problem by relaxing the requirement for annotated MRs, instead training the parser using indirect supervision of only the expected result of executing an MR corresponding to user's intent (i.e., an answer for a user question) (Clarke et al. 2010;Berant et al. 2013;Pasupat and Liang 2015). Compared with annotation-intensive MRs, such expected results are easy to obtain, even from annotators who do not know the specific MR language. Still, weakly supervised learning of semantic parsers remains a non-trivial task. First, the search space of possible MRs given an execution result is exponentially large, while the reward the parser receives during training is sparse and binary (e.g. one iff the execution result is correct); Another issue is spurious MRs, which are programs that happens to execute to the correct result, but are semantically incorrect. For example, a parser may wrongly parse the utterance "multiply two by two" to "2+2" and still get the correct answer. Such spuriousness would add a great amount of noise to the already-weak supervision signal.
Merging Weak and Active Supervision for Semantic Parsing
To tackle these challenges with weakly-supervised semantic parsing, we propose a method to combine Weak and Active Supervision for Semantic Parsing (WASSP). As illustrated in Fig. Fig. 1, for each iteration, we first train the semantic parsing model to convergence with weak supervision. Then we perform active learning which allows the model to actively ask for extra supervision (e.g. annotated MRs) for only the small fraction of examples that would " ! # "" # $! * 1 0 %! , -2 1 $6 , #" '-2 * * 1 0 %! ,-72 1 $6 , #" '-2 * * 1 0 %!2 1 $6 , #" '-2 * * #" ' ! " 37 % ) * #" ' $ (  7   :;  9<  9=  8   8<  89  7<  9   8;  7=  8;  : 7> 7< 86 * " ! 1 2 # #! ! # ' ! 1 2 . #" ' ! " 37 % ) / &" # $! 1 2 1 + 2 1 + 2 " $ ' " # # #! ! # " " # Figure 1: An overview of our proposed WASSP: we first train a semantic parser with weak supervision till convergence, then a small fraction of NLs are selected from the training set to actively query the annotator for extra supervision, which is then used to refined the candidate MRs for the next round of weakly-supervised training.
Merging Weak and Active Supervision for Semantic Parsing
maximally improve its performance. Finally, we feed the extra supervision to the semantic parsing model to rule out spurious logical forms and guide the search for hard examples. Given our proposed WASSP framework, we examine three main research questions: RQ1 How effective is WASSP in improving the performance of weakly-supervised semantic parser? RQ2 How can we pick the set of utterances where direct supervision will maximally benefit parsing accuracy? RQ3 What kind of extra supervision can WASSP use to improve the model for each sampled utterance? Specifically, to answer RQ1, we measure the improvement of WASSP over a weakly-supervised baseline with varying number of queries made. To answer RQ2, we investigate different heuristics to select a smallest set of NL utterances for annotation, by considering the informativeness, representativeness and diversity of an example. To answer RQ3, we study various forms of extra supervision to provide to the model for each sampled utterance. While just presenting the corresponding fully-specified MRs would be a trivial strategy, we show that this is not our only choice. Instead, using a simpler form, the sketch of an MR, the actively-supervised semantic parsers can achieve similar performance compared to using the resource-heavy MRs.
Merging Weak and Active Supervision for Semantic Parsing
We evaluate the effectiveness of our proposed active learning approach on two different datasets: the WikiSQL dataset (Zhong, Xiong, and Socher 2017) and the Wik-iTableQuestions dataset (Pasupat and Liang 2015). We demonstrate that active learning can greatly improve the performance semantic parsers trained with weak supervision with only a small fraction of examples actively selected and queried: By querying a total fraction of 1.8% of the training examples for WikiSQL, we improve over a stateof-the-art weakly-supervised baseline by 6.4%, achieving an accuracy of 79.0%, which is only 1.3% away from the model trained with full supervision. Experiment results on WikiTableQuestions show that WASSP can boost the performance of a model suffered from a cold start by 25.6% with only 50 examples. We also show that alternative forms of extra supervision to fully-specified MRs as MR sketches may also be used by WASSP and yields similar performances, only lose an maximum of 1.4% of accuracy comparing to full gold MR annotations on the WikiSQL dataset.
Merging Weak and Active Supervision for Semantic Parsing
Formally, given a natural language utterance q, a semantic parser is a parametric model P (z|q; θ) that transduces the utterance q into a meaning representation z. The parser could be optimized using either classical supervised learning, or weakly supervised learning.
Merging Weak and Active Supervision for Semantic Parsing
In the classical setting of (fully-) supervised learning, a model is optimized using a parallel corpus D consisting of paired NL utterances and their annotated MRs, i.e., D =
Merging Weak and Active Supervision for Semantic Parsing
The standard maximum likelihood learning problem is:
Merging Weak and Active Supervision for Semantic Parsing
Weakly-supervised learning can be formulated as a reinforcement learning problem, instead of access to goldstandard MRs, the semantic parser (i.e., the agent) is only presented with an execution model k (i.e., the environment, such as a database) and the gold execution result a. The parser interacts with k, searching for high-reward MRsẑ that execute to the correct results (i.e., k(ẑ) = a). A training instance is therefore defined by a triplet: utterance q, execution environment k and gold answer a. The learning objective is to maximize the probability of the correct answer a by marginalizing over all candidate MRs that execute to a:
Merging Weak and Active Supervision for Semantic Parsing
Weakly supervised parsers are usually trained with an EMlike method: the model searches for high-reward MRs by sampling from the semantic parser, then the model parameters are optimized to maximize the probability of generating these high-reward MRs.
Merging Weak and Active Supervision for Semantic Parsing
Two challenges make weakly-supervised parsing harder than its fully-supervised counterpart: Exploration of an Exponentially-large Space. The search space of latent MRs is exponential or infinite, making it intractable to exhaustively search over the set of plausible MRs {ẑ i |k(ẑ i ) = a i } to calculate sum in Eq.
Merging Weak and Active Supervision for Semantic Parsing
(2). Thus, it is common to resort to sampling or k-best search to approximate this space (Guu et al. 2017), potentially combined with sophisticated methods to reduce the space of plausible MRs with type constraints (Krishnamurthy, Dasigi, and Gardner 2017) or use of a memory buffer to cache high-reward explored MRs (Liang et al. 2018). However, inferring MRs for complex, highly compositional input utterances still remains an open challenge (Pasupat and Liang 2016). Spurious MRs. In contrast with the full-supervised setting where the parser is optimized using the semantically correct gold-standard MR for each utterance, the search space of latent MRs contains spurious samples that happen to execute to the correct results despite being semantically incorrect. For example, in Fig. 1, the NL utterance "Which country is the #1 gold medal winner?" has one semantically correct MR z 0 while MRs z 1 and z 2 do not match the semantics of the original utterance. Nonetheless, z 1 and z 2 follow the grammatical rules, satisfy the type constraints, and get correct results, and thus there is no simple way for a weakly supervised learner to distinguish them from gold-standard MRs. Such spurious MRs could be exponentially many (Pasupat and Liang 2016), and add significant noise to the training process. While attempts have been made to mitigate the issue of spuriousness, such as using a ranker trained with extra semantic or lexicon information (Cheng and Lapata 2018) or introduce prior knowledge to bias the policy (Misra et al. 2018), it is still highly non-trivial to totally solve this issue due to the ambiguous nature of the reward. Empirically, these challenges lead to performance gap between fully-and weakly-supervised semantic parsers. For example, on the WikiSQL dataset, the state-of-the-art fullysupervised semantic parser reaches an testing execution accuracy of 86.2% (Hwang et al. 2019) while the best weaklysupervised semantic parser (Liang et al. 2018) only has an accuracy of 72.6%, a difference of 13.6%.
Merging Weak and Active Supervision for Semantic Parsing
In order to solve these two problems, we first discuss three key insights about weakly-supervised semantic parsing that motivates WASSP: 1) Though mixed with spurious MRs, weakly-supervised semantic parsers are still able to find gold MRs for simpler input utterances, even with such noisy and weak learning signal. This means that effective training of semantic parsers does not require full supervision using annotated MRs for all examples.
Merging Weak and Active Supervision for Semantic Parsing
2) The EM-like weakly-supervised optimization will "stall" at the point when no new high-reward MRs can be found by the parser, and the set of discovered MRs with correct answers are unchanged for all examples in the training set. Eq.
Merging Weak and Active Supervision for Semantic Parsing
(2) will then converge to a local optimum. However, if the parser can discover new MRs with correct execution results, growing the set of high-reward MRs for even a small fraction of training examples, the optimization of Eq. (2) will resume, and the parser could in turn explore more similar high-reward MRs with its updated parameters. In other words, it is possible that some extra supervision for only a small fraction of the training data can prevent stagnation and resume the optimization process.
Merging Weak and Active Supervision for Semantic Parsing
3) Fully-specified MRs might not always be necessary to provide extra supervision for a weakly-supervised parser to discover high-reward MRs for complex utterances or rule out spurious MRs for ambiguous inputs. As an example, to identify the semantically correct MR for the illustrative example in Fig. 1, the parser only need to be informed that the gold MR should contain a superlative operation (i.e., argmax). This motivates us to explore alternative forms of extra supervision other than fully-specified MRs.
Merging Weak and Active Supervision for Semantic Parsing
Based on these three insights, we propose WASSP, a framework to merge active and weak supervision for semantic parsing. Fig. 1 presents a schematic overview. Specifically, WASSP iteratively performs the following steps:
Merging Weak and Active Supervision for Semantic Parsing
Step 1: Train the semantic parser P (ẑ|q; θ) to convergence by optimizing Eq. (2) over the training set D and discovered gold MR candidatesẐ;
Merging Weak and Active Supervision for Semantic Parsing
Step 2: With active sample selection heuristic, select a small subset of the training set D ⊆ D within certain budget and query the annotator for extra supervision for examples in D ;
Merging Weak and Active Supervision for Semantic Parsing
Step 3: After receiving annotations, e.g., in the form of annotated MRs {z * j |(k j , q j , a j ) ∈ D }, update the set of highreward MRsẐ explored so far for each training instance in D . If the form of extra supervision is annotated MRs, this simply amounts to setẐ to only contain the annotated MR, i.e.,Ẑ = {z * }. We will introduce other form of extra supervision in later sections.
Merging Weak and Active Supervision for Semantic Parsing
The motivation behind WASSP is very simple; every time weakly-supervised training stalls, WASSP allows the parser to select a small fraction of the training set and query for extra supervision, and use the received extra supervision to help continue the training process. In the following sections, we examine two design decisions: (1) how to select our subset of data to query D , and (2) what varieties of supervision can WASSP utilize.
Merging Weak and Active Supervision for Semantic Parsing
In this section, we examine strategies to select examples that are more likely to improve the performance of the semantic parser. First we propose a very simple yet effective correctness-based method, then we investigate two types of commonly adopted heuristics in active learning, uncertainty-based methods and coverage-based methods.
Merging Weak and Active Supervision for Semantic Parsing
In traditional pool-based active learning (Lewis and Gale 1994), the learner selects from a pool of examples without any information about the true label of example. However, for weakly-supervised semantic parsing we do have the expected execution result a as indirect information that can be used to inform our choice of data to annotate.
Merging Weak and Active Supervision for Semantic Parsing
As noted above, an MR executing to the correct result does not necessarily entail that the final logical form is correct (k(ẑ) = a ⇒ẑ = z) because the MR may be spurious. However, if an MR does not execute to the correct result, this does entail that the MR is incorrect (k(ẑ) = a ⇒ẑ = z). Taking advantage of this fact, we can derive a correctnessbased heuristic, which prefers to select the examples q for which there is no MR generated by the parser that matches the expected execution result thus no learning signal (i.e. reward) is provided for the semantic parser. These examples are much more likely to benefit from extra supervision than examples where the parser already has found at least one semantic parse corresponding to a correct answer.
Merging Weak and Active Supervision for Semantic Parsing
Uncertainty-based methods are a standard set of active learning techniques (Lewis and Catlett 1994;Tong and Koller 2001;Shen et al. 2004) that attempt to maximize the informativeness of selected examples by preferring examples where the learner is most uncertain about its prediction. In the context of semantic parsing, we follow (Duong et al. 2018) and measure the uncertainty on an example q by the least confidence score (Culotta and McCallum 2005):
Merging Weak and Active Supervision for Semantic Parsing
Though we can not enumerate all z in the MR space Z, it is easy to approximate this measurement by only enumerating all MRs remain in the beam Z B after beam search.
Merging Weak and Active Supervision for Semantic Parsing
Coverage-based methods are another common family of methods in active learning (Dasgupta and Hsu 2008). This type of methods consider the representativeness of the selected examples and attempts to cover as many other unselected examples as possible. Here we explore two different kinds of methods that adopt the idea of representativeness: Failed Word Coverage: Some particular words may be particularly difficult for semantic parsers to recognize, either because they are infrequent, or because they are only loosely connected to the MR. Based on this fact, we introduce a heuristic to select the examples with the largest number of words that are more prone to cause failure. An example is labeled as fail 
Merging Weak and Active Supervision for Semantic Parsing
In addition to the trivial solution of using fully-specified MRs as extra supervision, here we introduce an example of other forms of annotations that WASSP can utilize by merging them with the weakly-supervised training: the MR sketch. An example of a fully-specified MR and its sketch are shown in Fig. 2. Fully-specified MRs Using fully-specified MRs as extra supervision for the selected query examples is an obvious strategy. We define the fully-specified MRs as a sequence of complete operations or function calls with all the variables or arguments and ready to be executed. An example of this is shown in Fig. 2 as z f . A fully-specified MR can simply mount the explored high-reward MRs and be directly it used for training. MR Sketches A sketch of an executable MR is the sequence of operators or function names that does not fill in variables or arguments. An example MR sketch is shown in Fig. 2 as z s . With MR sketch annotation, we can: 1) Remove spurious high-reward MRs where the sketch does not match the gold MR sketch fromẐ; 2) Use this sketch as a guide for future exploration (e.g. constrained decoding), the high-reward MRs are only saved inẐ if their MR sketches match the gold MR sketch. These sketches provide several benefits: 1) Reduction of search space. Though MR sketches do not completely remove ambiguity, with the sketch, the parser only needs to fill out variables, and thus the search space is also greatly reduced. Moreover, since most neural semantic parsing models adopt a copy mechanism, variable type matching, or syntactic constraints (Krishnamurthy, Dasigi, and Gardner 2017;Liang et al. 2016), the search space becomes even more restricted and leaves little room for spurious MRs making it much easier to explore. For example, all spurious forms are pruned when given the gold MR sketch of (argmax ...)(hop ...) in Fig. 1; 2) Increased generality over the MR. Compared to annotating with fully-specified MRs (as z f in Fig. 2) is a higher-level abstraction, which resembles procedural knowledge for the model to parse this type of NL utterances without being deeply coupled with the specific details of the example itself. Take the example in Fig. 2, after annotating this example with its MR sketch (argmax ...)(hop ...), annotators can easily annotate other similar NL queries as "Which country has the largest population" or "Which team was ranked the last" without wasting time filling the detailed arguments. This special trait may speed up the manual annotation process thus could potentially be cheaper to obtain.
Merging Weak and Active Supervision for Semantic Parsing
Dataset: We evaluate the performance of WASSP on two different datasets: WikiSQL (Zhong, Xiong, and Socher 2017) and WikiTableQuestions (Pasupat and Liang 2015). The statistics of these two datasets are shown in Table 1. The WikiSQL dataset provides both gold MRs and their expected execution results, making it possible to perform simulated experiments where we train the same model with weak, active, or full supervision. WikiTableQuestions does not have annotated MRs, and thus we perform only limited active learning experiments querying real human annotators to add additional MRs. Neural Semantic Parsing Model: For the underlying neural semantic parsing model for WASSP, we adopt neural symbolic machines (NSM) ), a strong weakly-supervised semantic parser. NSM uses a sequenceto-sequence network to transduce an input utterance into an MR. The recurrent decoder is augmented with a memory component to cache intermediate execution results in a partially generated MR (e.g., the result of argmax for z 0 in Fig. 1 is cached as v 0 ), which can be referenced by following operations (e.g., in the hop function in z 0 ). We follow (Liang et al. 2018), and train NSM using memoryaugmented policy optimization (MAPO), which uses a memory buffer to cache currently explored high-reward MRs.
Merging Weak and Active Supervision for Semantic Parsing
The NSM trained with MAPO is the state-of-the-art model for weakly-supervised semantic parsing on the Wik-iSQL dataset (detailed below); a single model (i.e. without ensemble) can reach an execution accuracy of 72.4% and 72.6% on the dev and test set, respectively. Upon inspection, we found that the syntax rules of NSM did not have full coverage of the SQL queries in the WikiSQL dataset, and we further augmented the coverage of the NSM syntax rules, Acc. NSM+MML  70.7 NSM+MAPO  72.6 Previous Full Supervision Methods Acc. STAMP (Sun et al. 2018) 74.4 TranX+AP (Yin and Neubig 2018) 78.6 Coarse2Fine (Dong and Lapata 2018) 78.5 TypeSQL+TC (Yu et al. 2018) 82.6 improving this to 75.2% dev accuracy and 75.6% test accuracy with pure weak supervision. On the WikiTableQuestions dataset, an ensemble of 10 such models achieves the state-of-the-art performance of 46.3% execution accuracy while the performance of a single model is 43.1%(±0.5%). We use this as a strong baseline which we aim to improve.
Merging Weak and Active Supervision for Semantic Parsing
Training Procedure: First we follow the procedure of (Liang et al. 2018) to train NSM with MAPO on both Wik-iSQL and WikiTableQuestions datasets with the same set of hyperparameters as used in the original paper. Then for WikiSQL, we run WASSP for 3 iterations with query budget 1,000 or more and only run for one iteration with smaller budget. For each iteration, the model queries for extra supervision and then it is trained for another 5K steps. The query budget is evenly distributed to these 3 iterations and limited by the total amount. For WikiTableQuestions, we simply run one such iteration (due to limit number of annotations obtained) but let it train for 50K steps with human annotated MRs. 
Merging Weak and Active Supervision for Semantic Parsing
Effectiveness of WASSP. First, to answer RQ1, we primarily investigate the performance of WASSP w.r.t. the amount of extra supervision during training. Table 2 and Table 3 list results of WASSP with varying query budgets on Wik-iSQL and WikiTableQuestions respectively. On the Wik-iTableQuestions dataset, MAPO has the option to begin with an empty memory buffer (i.e. a cold start) or warm up the memory buffer by searching with manually-designed prun- 
Merging Weak and Active Supervision for Semantic Parsing
Acc. (Pasupat and Liang 2015) 37.1 (Neelakantan et al. 2016) 34.2 (Haug, Ganea, and Grnarova 2018) 34.8 (Zhang, Pasupat, and Liang 2017) 43.7 Table 3: WASSP with varying budget and different types of weakly-supervised training start on WikiTableQuestions. Evaluated by execution accuracy on the test set, improvements are noted in brackets.
Merging Weak and Active Supervision for Semantic Parsing
ing rules as heuristics (i.e. a warm start). Since designing the pruning heuristics also includes non-trivial human effort, we include the both results from a model trained from a cold start and a warm start to have a rather complete study. Our model presented here uses the correctness-based query selection heuristic and fully-specified MRs as extra supervision. We also compare WASSP with existing weakly-or fully-supervised methods.
Merging Weak and Active Supervision for Semantic Parsing
From Table 2 we can see that with annotating only 100 (0.2%) examples from the training set, WASSP can improve the performance by 0.7% while querying 500 examples (0.89%) achieves an absolute improvement of 2.1% over the same model trained with pure weak supervision. Notably, if we allow a query budget of 1,000 examples, WASSP achieves an execution accuracy of 78.6%, a 6.0% absolute improvement over previous state-of-the-art result (72.6%) with only weak supervision (Liang et al. 2018). Additionally, with only 1.8% of annotated training examples, WASSP is already quite competitive against some of the strong fully-supervised systems trained with the whole annotated training set (Sun et al. 2018;Yin and Neubig 2018;Dong and Lapata 2018). This result is also only 1.7% away from the same semantic parser (NSM) trained with full supervision (80.3%). Finally, if we further increase the query budget to 10,000 examples, WASSP could achieve similar performance comparable with training the semantic parser using full supervision, but with 80% less annotated data. This set of results shows that WASSP allows learning with significantly less supervision.
Merging Weak and Active Supervision for Semantic Parsing
We also have some interesting findings for the WikiTable-Questions dataset. While Table 3 shows that WASSP can improve the performance of a model trained with either cold or warm start, WASSP yields a substantial on models trained with a cold start: a 25.6% improvement with only 50 examples. Further investigations shows that the baseline model trained with a cold start on WikiTableQuestions is only able to correctly parse simple NL queries (e.g. count the number of rows, find maximum number, etc) that can be addressed with one statement (e.g.(count ...), (maximum ...)). Since the model is under-trained, the probability of it exploring a high-reward complex MR for a sophisticated questions is very low. In this way, injecting a  Table 4. We found the simple correctness-based heuristic is effective, perhaps surprisingly so. Using uncertainty scores alone does not achieve good performance, barely exceeding random selection. Further investigation finds that by the end of weakly-supervised training, 84.0% of examples in training set with their parsed MRs not executed to the expected results have a confidence (Eq. (3)) over 0.9. This finding suggests that incorrect MRs could still have high modelassigned probability, which is worth future investigation.
Merging Weak and Active Supervision for Semantic Parsing
Nevertheless, WASSP achieves the best performance by combining the two heuristics (i.e., selecting the most uncertain examples among the failed ones) 2 . This query sample selection method combines the idea of correctness and uncertainty to maximizes the informativeness of the selected examples and reaches 78.4% and 79.0% of execution accuracy on dev. and test set. Finally, we found the coveragebased methods do not work as well, which may result from our relatively simple approach of measuring coverage using sentence embedding or failed words. 3 Forms of Supervision Here we study how well WASSP can adapt to other forms of supervision besides the gold fullyspecified MR. To answer RQ3, we conduct experiments with the MR sketch as extra supervision and compare its performance to using the fully-specified MR under the framework of WASSP and the results are shown in Table 5 and Table 6 for the WikiSQL and WikiTableQuestions datasets, respectively. In this set of experiments, we annotate examples selected with correctness heuristic.
Merging Weak and Active Supervision for Semantic Parsing
From these tables, we can see that the performance of a model learnt from MR sketches as extra supervision is comparable to the model learnt from fully-specified MR, 
Merging Weak and Active Supervision for Semantic Parsing
Here we conduct a qualitative analysis on how the examples selected and queried by WASSP help it generalize to other unseen or not queried examples. Example 1 from Fig. 3 shows that extra supervision for the query set can generalize to other examples with similar NL utterance structure. For Example 2, the missing filter is corrected after receiving extra supervision for an example with similar conditions. As in Example 3, the parser can not use the correct filter for comparing time, but as the example above being selected and queried, the semantic parser trained with extra supervision learns to map "later than" to filter greater for com- Related Work (Artzi and Zettlemoyer 2013) proposed to jointly learn from meaning and context for. In the work of (Krishnamurthy and Mitchell 2012), they combined weak supervision from the knowledge base and the NL sentences. To handle the problem of spurious forms, (Muhlgay, Herzig, and Berant 2018) proposed a value-based search method with a trained critic network trained with the environment. In (Shen et al. 2004), they explored different active learning heuristics for named entity recognition, including informativeness, representativeness and diversity. (Duong et al. 2018) studied active learning for fully-supervised semantic parsing and showed that an uncertainty-based active learning method is powerful for traditional data collection but not useful to overnight data collection.
Merging Weak and Active Supervision for Semantic Parsing
We propose WASSP, a framework to merge weak and active learning for semantic parsing. We study different query sample selection heuristics and various forms for extra supervision. Experiments on two datasets show that WASSP can greatly improve the performance of a weakly-supervised semantic parser with a small fraction of examples queried.
Active Learning for Multilingual Semantic Parser
Multilingual semantic parsing converts multilingual natural language utterances into logical forms (LFs) using a single model. However, there is a severe data imbalance among the MSP datasets. Currently, most semantic parsing datasets are in English, while only a limited number of non-English datasets exist. To tackle the data imbalance issue, almost all current efforts build MSP datasets by translating utterances in the existing datasets from the resource-rich language (e.g. English) into other languages (Duong et al., 2017;Li et al., 2021a). However, manual translation is slow and laborious. In such cases, active learning is an excellent solution to lower the translation cost.
Active Learning for Multilingual Semantic Parser
Active learning (AL) is a family of methods that collects training data when the annotation budgets are limited (Lewis and Catlett, 1994). Our work proposes the first active learning approach * Most of this author's work was completed during his internship at Openstream.AI. for MSP. Compared to translating the full dataset, AL-MSP aims to select only a subset from the existing dataset to be translated, which significantly reduces the translation cost.
Active Learning for Multilingual Semantic Parser
We further study which examples AL-MSP should select to optimize multilingual parsing performance. Oren et al. (2021) demonstrated that a training set with diverse LF structures significantly enhances compositional generalization of the parsers. Furthermore, our experiments show that the examples with LFs aligned with more diversified lexical variants in the training set considerably improve the performance of multilingual parsing during AL. Motivated by both, we propose a novel strategy for selecting the instances which include diversified LF structures with more lexical choices. Our selection method yields better parsing performance than the other baselines. By translating just 32% of all examples, the parser achieves comparable performance on multilingual GEO-QUERY and NLMAP as translating full datasets.
Active Learning for Multilingual Semantic Parser
Prior works obtain the hyperparameters of the AL methods by either copying configurations from comparable settings or tuning the hyperparameters on the seed evaluation data (Duong et al., 2018). However, the former method is not suitable as our AL setting is unique, whereas the second method requires extra annotation costs. In this work, we provide a cost-free method for our AL scenario for obtaining optimal hyperparameters.
Active Learning for Multilingual Semantic Parser
Our contributions are i) the first active learning procedure for MSP that reduces the translation effort, ii) an approach that selects examples for getting superior parsing performance, and iii) a hyperparameter tuning method for the selection that does not incur any extra annotation costs.
Active Learning for Multilingual Semantic Parser
Multilingual Semantic Parsing. A multilingual semantic parser is a parametric model P θ (y|x) that estimates the probability of the LF y ∈ Y condi-tioned on the natural language utterance x ∈ X l in an arbitrary language from a language set l ∈ L. The model is trained on the utterance-LF pairs Atoms and Compounds. Each logical form sequence can be represented as a semantic tree, y = τ y . Oren et al. (2021); Shaw et al. (2021) define the nodes and sub-trees in τ y as the atoms and compounds, respectively. Increasing the diversity of the atoms and compounds in the training set improves the parser's compositional generalization (Oren et al., 2021;Li et al., 2021b). For example, an LF "( lambda $0 e ( and ( ground_transport $0 ) ( to_city $0 ci0 ) ) )" can be expressed as a tree as in Fig. 1. The atoms are nodes such as "lambda", "$0", "e" in the LF tree. In this work, the compounds are defined as two-level sub-trees such as "( ground_transport $0 )", "( to _city $0 ci0 )", "( and ground_transport to_city )", and "( lambda $0 e and )" in the LF tree. Data Collection for MSP. Prior data collection or active learning works annotates the LFs for the utterances (Duong et al., 2018;Sen and Yilmaz, 2020) or vice versa (Duong et al., 2018;Wang et al., 2015). But most MSP works (Susanto and Lu, 2017;Li et al., 2021a) obtain data by translating existing datasets from high-resource languages into low-resource languages, which is less costly since it does not need annotators' expertise in LFs. Following the same annotation paradigm, our AL does not annotate LFs for multilingual utterances, but instead chooses the utterances to be translated.
Active Learning for Multilingual Semantic Parser
AL-MSP considers only a bilingual scenario for the proof of concept, while extending our AL method to more than two languages is easy. The goal of AL-MSP is to minimize the human effort in translating utterances while the semantic parser can still achieve a certain level of performance on the bilingual test sets. Starting from a semantic parser initially trained on the dataset
Active Learning for Multilingual Semantic Parser
The selection criterion is based on our proposed acquisition function ϕ(e s ) scoring each example, e s = (x s , y). The parser is re-trained on the union ofD t and D s . There will be Q iterations of selection and re-training until the re-trained parser reaches a good performance on the bilingual test sets T s and T t . Algorithm 1 describes our experimental settings in detail.
Active Learning for Multilingual Semantic Parser
Input :Initial training set D 0 = D 0 s , budget size Kq, number of the selection rounds Q Output :A well-learned multilingual parser P θ (y|x) Train the parser P θ (y|x) on the training set D 0 for q ← 1 to Q do Estimate the acquisition ϕ(·) Select a subsetD q s ∈ D q−1 s of the size Kq based on the acquisition function ϕ(·) Translate the utterances inD q s into the target language,D q t = ts→t(D q s ). Combine the training sets,
Active Learning for Multilingual Semantic Parser
Re-train the parser P θ (y|x) on D q Evaluate parser performance on test sets Ts, Tt end
Active Learning for Multilingual Semantic Parser
Our selection strategy selects the untranslated examples which maximize the acquisition scores. The acquisition comprises two individual terms, LF Structure Diversity and Lexical Choice Diversity. LF Structure Diversity (LFSD). We give a simple technique to diversify the LF substructures (atoms and compounds) in the instances. At qth iteration, let D l s = q−1 i=1D i s denotes all the translated examples and D u s = D q−1 s be the untranslated ones. We partition their union D u s ∪ D l s into |D l s | + K q clusters with Incremental K-means (Dataiku Lab, 2022). Each example e s = (x s , y) is featurized by extracting all the atoms and compounds in the LF tree τ y , followed by calculating the TF-IDF (Salton and McGill, 1986) value for each atom and com-pound. Incremental K-means considers each example of D l s as a fixed clustering centroids and estimates K q new cluster centroids. For each of the K q new clusters, we select one example closest to the centroid.
Active Learning for Multilingual Semantic Parser
Such selection strategy is reformulated as selecting K q examples with the highest acquisition scores one by one at each iteration:
Active Learning for Multilingual Semantic Parser
where f (·) is the feature function, m(·) maps each LF into its cluster id and c i is the center embedding of the cluster i. As in Algo. 1, when a new example is chosen, none of its cluster mates will be selected again. The incremental mechanism guarantees the newly selected examples are structurally different from those chosen in previous iterations. Since we use batch-wise AL, we just estimate the clusters once per iteration to save the estimation cost. Lexical Choice Diversity (LCD). LCD aims to select examples whose LFs are aligned with the most diversified lexicons. We achieve this goal by choosing the example maximizing the average entropy of the conditional probability p(x s |a):
Active Learning for Multilingual Semantic Parser
where a is the atom/compound, A y is the set of all atoms/compounds extracted from y, V s is the vocabulary of the source language, A l is the set of atoms/compounds in all selected examples until now, and p(x s |a) is constructed by counting the cooccurrence of a and x s in the source-language training set. To prevent selecting structurally similar LFs, the score of each selected atom or compound is penalized by a decay weight β. Our intuition has two premises. First, the parser trained on example pairs whose LFs have more lexical choices generalizes better. Second, LFs with more source-language lexical choices will have more target-language lexical choices as well. LF Structure and Lexical Choice Diversity (LFS-LC-D). We eventually aggregate the two terms to get their joint benefits, ϕ(x s , y) = αϕ s (x s , y) + ϕ c (x s , y), where α is the weight that balances the importance of two terms. We normalize the two terms using quantile normalization (Bolstad et al., 2003) in order to conveniently tune α. Hyperparameter Tuning. Because our setup is unique, we can not copy hyperparameters from existing works. The other efforts (Duong et al., 2018) get hyperparameters by evaluating algorithms on seed annotated data. To tune our AL hyperparameters, α and β, a straightforward practice using seed data is to sample multiple sets of examples from the source-language data, the target-language counterparts of which are in seed data, by varying different hyperparameter configurations and reveal their translations in the target language, respectively. The parser is trained on different bilingual datasets and evaluated on the target-side dev set. We use the one, which results in the best parsing performance, as the experimental configuration.
Active Learning for Multilingual Semantic Parser
Such a method still requires translation costs on the seed data. We assume if the selected examples help the parser generalize well in parsing sourcelanguage utterances, their translations should benefit the parser in parsing target languages. Given this assumption, we propose a novel cost-free hyperparameter tuning approach. First, we acquire different sets of source-language samples by varying hyperparameters. Then, we train the parser on each subset and evaluate the parser on the sourceside dev set. Finally, we use the hyperparameters with the best dev set performance.
Active Learning for Multilingual Semantic Parser
Datasets. We experiment with multilingual GEO-QUERY and NLMAP. GEOQUERY utterances are in English (EN), German (DE), Thai (TH), and Greek (EL); NLMAP utterances are in English and German. Neither corpora include a development set, so we use 20% of the training sets of GEO-QUERY and NLMAP in each language as the development sets for tuning the hyperparameters. To simulate AL process, we consider English as the resource-rich language and others as the target languages. After the examples are selected from the English datasets, we reveal their translations in the target languages and add them to the training sets. AL Setting. We perform six iterations, accumulatively selecting 1%, 2%, 4%, 8%, 16% and 32% of examples from English GEOQUERY and NLMAP. Baselines. We compare four selection baselines and the oracle setting: i) Random picks English utterances randomly to be translated, ii) S2S (FW) (Duong et al., 2018) selects examples with the lowest parser confidence on their LFs, iii) CSSE (Hu and Neubig, 2021) selects the most representative and diversified utterances for machine translation, iv) Max Compound (Oren et al., 2021) selects examples that diversify the atoms and compounds in the LFs, v) ORACLE trains the parser on the full bilingual training set. Evaluation. We adopt the exact match accuracy of LFs for all the experiments. We only report the parser accuracy on the target languages as we found the influence of new data is negligible to the parser accuracy on English data (See Appendix A.2). Base Parser. We employ BERT-LSTM (Moradshahi et al., 2020) as our multilingual parser. Please see Appendix A.1 for its detailed description. Table 1 displays the experiment results with the hyperparameters tuned using only English data (EN) and the hyperparameters tuned using seed data on i) English data plus a small subset (10% of train data plus development data) in the target language (EN + 10%), ii) the full bilingual data (EN + full), iii) the same dataset in a different pair of languages from our experiment languages (Diff Lang), iv) a different dataset in the same languages as our experiment (Diff Data). From Table 1, we can see our approach takes significantly fewer annotation resources than others to find optimum hyperparameters. Adding more target-language data does not help obtain better hyperparameters, validating our assumption that English data is enough for LFS-LC-D to obtain good hyperparameters. Surprisingly, the hyperparameters tuned on a different language pair do not significantly worsen the selection choices. However, tuning hyperparameters from other datasets results in inferior parsing performance, which is anticipated as different datasets include different LFs, but the performance of LFS-LC-D is closely related to the LF structures.
Active Learning for Multilingual Semantic Parser
Effectiveness of AL-MSP. Fig. 2 shows that only a small amount of target-language data significantly improves the parsing performance over the zero-shot performance. For example, merely 1% of training data improves the parsing accuracies by up to 13%, 12%, 15% and 6% on GEO-QUERY(DE), GEOQUERY(TH), GEOQUERY(EL) and NLMAP(DE), respectively. With the best selection approach LFS-LC-D, translating 32% of instances yields parsing accuracies on multilingual GEOQUERY and NLMAP that are comparable to translating the whole dataset, with an accuracy gap of less than 5%, showing that our AL-MSP might greatly minimize the translation effort. Effectiveness of LFS-LC-D. LFS-LC-D consistently outperforms alternative baselines on both multilingual datasets when the sampling rate is lower than 32%. In contrast, S2S(FW) consistently yields worse parser performance than the other baselines. Our inspection reveals that the parser is confident in instances with similar LFs. MAX COMPOUND diversifies LF structures as LFS-LC-D, however it does not perform well on GEOQUERY(TH). CSSE diversifies utterances yet performs poorly. We hypothesize that diversifying LF structures is more advantageous to the semantic parser than diversifying utterances. RANDOM also performs consistently across all settings but at a lesser level than LFS-LC-D. Individual Terms of LFS-LC-D. We also inspect each individual term, LFSD and LCD, in LFS-LC-D. As in Fig. 3, both terms have overall lower performance than LFS-LC-D, indicating the combination of two terms is necessary. Specifically, LFSD performs poorly on NLMAP at the low sampling region. We inspect that NLMAP includes 5x more compounds than GEOQUERY. Therefore, it is difficult for the small number of chosen examples to encompass all types of compounds. LCD performs poorly on GEOQUERY(TH). We notice that Thai is an analytic language linguistically distinct from English, German or Greek, so the entropy values of the probability p(x s |a) over lexicons in Thai (p=0.03) is statistically more different to the ones over English than German (p=5.80e-30), and Greek (p=1.41e-30) 1 . Overall, the two terms could  benefit each other, so LFS-LC-D performs steadily across different settings. Comparison with Machine Translation. We also evaluate the parsers that utilize machine translation services. The parsers are trained on a combination of English data and data translated into the target language by Google Translation (Wu et al., 2016). The accuracy of parsers evaluated on test sets of Geo(De), Geo(Th), Geo(El), and NlMap(De) was 49%, 58%, 75%, and 75%, respectively. These parsing accuracies are significantly lower than those attained by parsers trained on data provided through human translation, which achieved 80%, 80%, 81%, and 83%, respectively. This suggests that the performance of the parser is tightly correlated to the quality of the employed machine translation system. Clearly, human translation delivers a greater output quality compared to machine translation. In addition, the results reveal that parsers employing AL methods can easily outperform those employing machine translation methods, particularly when the sampling rate for AL is more than 1%, 4%, 8%, and 32% in the four data settings.
Active Learning for Multilingual Semantic Parser
We conducted the first in-depth empirical study to investigate active learning for multilingual seman-tic parsing. In addition, we proposed a method to select examples that maximize MSP performance and a cost-free hyperparameter tuning method. Our experiments showed that our method with the proper hyperparameters selects better examples than the other baselines. Our AL procedure with the ideal example selection significantly reduced the translation effort for the data collection of MSP.
Active Learning for Multilingual Semantic Parser
To reduce annotation costs, existing data collection methods for MSP also utilize machine translation (Moradshahi et al., 2020). Despite the generally lower quality of machine-generated translations compared to human translations, the cost of machine translation services is notably more economical. Our study pioneers the investigation into the feasibility of reducing annotation costs by manually translating only selective portions of the utterance pool. In our work, we provide an initial evaluation of parsers using machine translation versus those using AL methods. Further research is necessary to thoroughly compare these cost-reduction approaches, highlighting their respective advantages and limitations, which we intend to pursue as part of our future work.
Active learning for deep semantic parsing
Semantic parsing maps a natural language query to a logical form (LF) Collins, 2005, 2007;Haas and Riezler, 2016;Kwiatkowksi et al., 2010). Producing training data for semantic parsing is slow and costly. Active learning is effective in reducing costly data requirements for many NLP tasks. In this work, we apply active learning to deep semantic parsing and show that we can substantially reduce the data required to achieve state-of-the-art results.
Active learning for deep semantic parsing
There are two main methods for generating semantic parsing training data. The traditional approach first generates the input natural language utterances and then labels them with output LFs. We show that active learning based on uncertainty sampling works well for this approach.
Active learning for deep semantic parsing
The "overnight" annotation approach (Wang et al., 2015) generates output LFs from a grammar, and uses crowd workers to paraphrase these LFs into input natural language queries. This approach is faster and cheaper than traditional annotation. However, the difficulty and cost of data generation and validation are still substantial if we need a large amount of data for the system to achieve high accuracy; if the logical forms can express complex combinations of semantic primitives that must be covered; or if the target language is one with relatively few crowd workers.
Active learning for deep semantic parsing
Applying active learning to the overnight approach is even more compelling, since the unlabelled LFs can be generated essentially for free by a grammar. However, conventional active learning strategies are not compatible with the overnight approach, since the crowd annotators produce inputs (utterances) rather than labels (LFs).
Active learning for deep semantic parsing
In order to apply active learning to deep semantic parsing, we need a way of selecting hyperparameters without requiring the full training dataset. For optimal performance, we should rerun hyperparameter tuning for each active learning round, but this is prohibitively expensive computationally. We show that hyperparameters selected using a random subset of the data (about 20%) perform almost as well as those from the full set.
Active learning for deep semantic parsing
Our contributions are (1) a simple hyperparameter selection technique for active learning applied to semantic parsing, and (2) straightforward active learning strategies for both traditional and overnight data collection that significantly reduce data annotation requirements. To the best of our knowledge we are the first to investigate active learning for overnight data collection.
Active learning for deep semantic parsing
Sequence-to-sequence models are currently the state-of-the-art for semantic parsing (Jia and Liang, 2016;Dong and Lapata, 2016;Duong et al., 2017). In this paper, we also exploit a sequenceto-sequence model to minimise the amount of la-belled training data required to achieve state-ofthe-art semantic parsing results.
Active learning for deep semantic parsing
Active learning has been applied to a variety of machine learning and NLP tasks (Thompson et al., 1999;Tang et al., 2002;Chenguang Wang, 2017) employing various algorithms such as least confidence score (Culotta and McCallum, 2005), large margin (Settles and Craven, 2008), entropy based sampling, density weighting method (Settles, 2012), and reinforcement learning (Fang et al., 2017). Nevertheless, there has been limited work applying active learning for deep semantic parsing with the exception of Iyer et al. (2017). Different from conventional active learning, they used crowd workers to select what data to annotate for traditional semantic parsing data collection.
Active learning for deep semantic parsing
In this paper, we apply active learning for both traditional and overnight data collection with the focus on overnight approach. In addition, a limitation of prior active learning work is that the hyperparameters are usually predefined in some way, mostly from different work on the same or similar dataset, or from the authors experience Fang et al., 2017). In this paper, we investigate how to efficiently set the hyperparameters for the active learning process.
Active learning for deep semantic parsing
We base our approach on the attentional sequenceto-sequence model (S2S) of Bahdanau et al. (2014). This attentional model uses a bidirectional recurrent neural network (RNN) to encode a source as a sequence of vectors, which are used by another RNN to generates output. Given the source utterance x = [x 1 , x 2 , ...x n ] and target LF y = [y 1 , y 2 , ...y m ], we train the model to minimize the loss under model parameters ✓.
Active learning for deep semantic parsing
Additionally, we apply the UNK replacement technique in Duong et al. (2017), keeping the original sentence in the data. 1
Active learning for deep semantic parsing
There is a diversity of strategies for active learning. A simple and effective active learning strategy is based on least confidence score (Culotta 1 We call S2S model applied to traditional data collection and overnight data collection as forward S2S and backward S2S respectively. The forward S2S model estimates P(y|x), the backward S2S model estimates P(x|y).
Active learning for deep semantic parsing
and McCallum, 2005). This strategy selects utterance x 0 to label from the unlabelled data U x as follows:
Active learning for deep semantic parsing
where y ⇤ is the most likely output. We found that this least confidence score works well across datasets, even better than more complicated strategies in traditional data collection (described below).
Active learning for deep semantic parsing
In the traditional (forward) approach, we start with the list of unlabelled utterances and an initial seed of utterances paired with LFs. We gradually select utterances to annotate with the aim of maximizing the test score as early as possible. We use forward S2S sentence loss as defined in Equation (1) as the least confidence score measurement (i.e. select the instance with higher loss). The drawback of a least confidence score strategy (and strategies based on other measurements such as large margin), is that they only leverage a single measurement to select utterances (Settles and Craven, 2008). To combine multiple measurements, we build a classifier to predict if the model will wrongly generate the LF given the utterance, and select those utterances for annotation. The classifier is trained on the data generated by running 5-fold cross validation on annotated data. 2 We exploit various features, including sentence log loss, the margin between the best and second best solutions, source sentence frequency, source encoder last hidden state and target decoder last hidden state (see supplementary material §A.1 for more detail) and various classifier architectures including logistic regression, feedforward networks and multilayer convolutional neural networks. On the development corpus, we observed that the least confidence score works as well as the classifier strategy.
Active learning for deep semantic parsing
In the overnight (backward) approach, we start with the set of all unlabelled LFs (U y ), and an initial randomly-selected seed of LFs paired with utterances (i.e. labelled LFs L y ). The aim is to select LFs for which we should obtain utterances, maximizing the test score as early as possible. In the overnight approach, we can't use the least confidence score (i.e. the forward S2S sentence loss) directly since we can't estimate P(y|x) because we don't know the utterance x. We have to somehow approximate this probability with regard to the performance on test.
Active learning for deep semantic parsing
A simple strategy is just to apply the backward S2S model and estimate P(x|y), e.g. we select LF y 0 to label from the unlabelled data U y as follows:
Active learning for deep semantic parsing
Essentially, we train the S2S model to predict the utterance given the LF. The motivation is that if we can reconstruct the utterance from the LF then we could possibly generate LFs from utterances. However, this strategy ignores one important aspect of semantic parsing, which is that LFs are an abstraction of utterances. One utterance is mapped to only one LF, but one LF corresponds to many utterances.
Active learning for deep semantic parsing
Since the forward S2S loss performs so well, another strategy is to approximate the selections made by this score. We train a linear binary classifier 3 to predict selections, using features which can be computed from LFs only. We extract two set of features from the LF model and the backward S2S model. The LF model is an RNN language model but trained on LFs (Zaremba et al., 2014). 4 We extract the LF sentence log probability i.e. log P(y), feature from this model. The backward S2S model, as mentioned above, is the model trained to predict an utterance given a LF. We extracted the same set of features as mentioned in §4.1 including LF sentence log loss, margin between best and second best solutions, and LF frequencies.
Active learning for deep semantic parsing
On the development corpus, we first run one active learning round using forward S2S model sentence loss (i.e. modelling P(y|x)) on the initial annotated data L y . The set of selected LFs based on forward S2S loss will be the positive examples, and all other LFs that are not selected will be the negative examples for training the binary classifier. Our experiments show that the classifier which uses the combination of two features (source LF frequencies and the margin of best and second best solution) are the best predictor of what is selected by forward S2S model log loss (i.e. modelling P(y|x)). It is interesting to see that absolute score of backward S2S model loss is not a good indicator as it is not selected. This may be due to the fact that utterance-LF mapping is oneto-many and the model probability is distributed to all valid output utterances. Hence, low probability is not necessary an indicator of bad prediction. We use the linear combination of the two features mentioned above with the weights from the binary classifier as a means of selecting the LF for overnight active learning on different corpora without retraining the classifier.
Active learning for deep semantic parsing
We experiment with the NLMaps corpus (Haas and Riezler, 2016) which was collected using the traditional approach. We tokenize following Kočiský et al. (2016). We also experiment with the Social Network corpus from the Overnight dataset (Wang et al., 2015) (which was collected using the overnight approach). Social Network was chosen as being the largest dataset available. Since neither corpora have a separate development set, we use 10% of the training set as development data for early stopping. We select ATIS (Zettlemoyer and Collins, 2007) as our development corpus for all feature selection and experiments with classifiers in §4.1 and §4.2.
Active learning for deep semantic parsing
For evaluation, we use full LF exact match accuracy for all experiments (Kočiský et al., 2016). Note that this is a much stricter evaluation compared with running through database evaluator as in Wang et al. (2015).
Active learning for deep semantic parsing
Hyperparameter tuning is important for good performance. We tune the base S2S model ( §3) on the development data by generating 100 configurations using Adam optimizer (Kingma and Ba, 2014) and a permutation of different source and target RNN sizes, RNN cell types, initializer, dropout rates and mini-batch sizes.
Active learning for deep semantic parsing
As mentioned, hyperparameter tuning is often overlooked in active learning. The common approach is just to use the configuration from a similar problem, from prior work on the same dataset, Random baseline randomly select the training data at each round. Fw S2S is used for traditional data collection using forward S2S loss score. Bw S2S is used for overnight data collection using backward S2S loss score. Bw classifier is used also for the overnight approach but linearly combines several scores together as mentioned in §4.  (Duong et al., 2017) and Social Network (Jia and Liang, 2016) are provided for reference. 5 or based on the authors own experience. However, in practice we don't have any prior work to copy the configuration from. Table 1 shows the experiments with the NLMap and Social Network corpora with configurations: 1) copied from another dataset (ATIS), 2) tuned on a small subset (10% of train data plus development data) and 3) tuned on the full dataset. We can see that copying from a different dataset results in a suboptimal solution, which is expected since the different datasets are significantly different. It is surprising that tuning on small subset of the data performs as well as tuning on all the data and, more importantly, it achieves similar results as the state of the art (SOTA). Figure 1 shows the active learning curve for NLMap, ATIS and Overnight (Social Network) datasets. 10% of data is randomly selected as initial seed data for active learning and hyperparameter tuning. We run active learning for 10 rounds, selecting 10% of the data at each round. Round 0 reports the result trained on the initial seed data and round 9 is the result on the whole training data. For reference, we also report Fw S2S for Social Network, treating that corpus as if they were collected using the traditional approach, and Bw S2S/classifier for NLMap and ATIS treating those corpora as if they were collected using the overnight approach. For traditional data collection (forward direction), S2S loss consistently outperforms the random baselines on both datasets. The differences are as high as 9% for NLMap (at round 4). Applying this strategy for ATIS, we reach SOTA results at round 4, using only 50% of data. We also experimented with the large margin baseline and classifier strategies as mentioned in §4.1. The least confidence strategy using S2S loss outperforms large margin and achieves similar performance with the more complicated classifier strategy, thus we omit those results for brevity.
Active learning for deep semantic parsing
On the overnight data collection active learning (backward direction), the results are split. The backward S2S loss performs particularly well on the NLMap corpus, approximating the forward S2S performance. However, it performs similar to the random baseline in the other corpora. On the other hand, the classifier strategy performs well on both ATIS and Social Network but poorly on NLMap. Using this strategy, we approximate the SOTA for both ATIS and Social Network at round 5 and 6 respectively (saving 40% and 30% of data). We suspect that backward S2S loss performs so well on NLMap since there is a one-toone mapping between utterance and LF. The number of unique LFs in the training data for NLMap, ATIS and Overnight are 95.4%, 28.4% and 19.5% respectively. All in all, our proposed strategies for "overnight" active learning are nearly as good as traditional active learning, showing in similar area under the curve value in Figure 1.
Active learning for deep semantic parsing
We have discussed practical active learning for deep semantic parsing. We have empirically shown that it is possible to get good hyperparameters from only a small subset of annotated data. We applied active learning for both traditional and overnight semantic parsing data collection. For traditional data collection, we show that least confidence score based on S2S log loss performs well across datasets. Applying active learning for overnight data collection is challenging, and the best performing strategy depends on the domain. We recommend that applications explore both the backward S2S and classifier strategies.