generative question answering
235313524
Can Generative Pre trained Language Models Serve as Knowledge Bases for Closed book QA?
Recent work has investigated the interesting question using pre trained language models (PLMs) as knowledge bases for answering open questions. However, existing work is limited in using small benchmarks with high test train overlaps. We construct a new dataset of closed book QA using SQuAD, and investigate the performance of BART. Experiments show that it is challenging for BART to remember training facts in high precision, and also challenging to answer closed book questions even if relevant knowledge is retained. Some promising directions are found, including decoupling the knowledge memorizing process and the QA finetune process, forcing the model to recall relevant knowledge when question answering.
ACL/IJCNLP
Cunxiang Wang Pai Liu
235294086
Multilingual Medical Question Answering and Information Retrieval for Rural Health Intelligence Access
In rural regions of several developing countries, access to quality healthcare, medical infrastructure, and professional diagnosis is largely unavailable. Many of these regions are gradually gaining access to internet infrastructure, although not with a strong enough connection to allow for sustained communication with a medical practitioner. Several deaths resulting from this lack of medical access, absence of patient's previous health records, and the unavailability of information in indigenous languages can be easily prevented. In this paper, we describe an approach leveraging the phenomenal progress in Machine Learning and NLP (Natural Language Processing) techniques to design a model that is low resource, multilingual, and a preliminary first point of contact medical assistant. Our contribution includes defining the NLP pipeline required for named entity recognition, language agnostic sentence embedding, natural language translation, information retrieval, question answering, and generative pre training for final query processing. We obtain promising results for this pipeline and preliminary results for EHR (Electronic Health Record) analysis with text summarization for medical practitioners to peruse for their diagnosis. Through this NLP pipeline, we aim to provide preliminary medical information to the user and do not claim to supplant diagnosis from qualified medical practitioners. Using the input from subject matter experts, we have compiled a large corpus to pre train and fine tune our BioBERT based NLP model for the specific tasks. We expect recent advances in NLP architectures, several of which are efficient and privacy preserving models, to further the impact of our solution and improve on individual task performance.
ArXiv
Vishal Vinod Susmit Agrawal Vipul Gaurav R Pallavi Savita Choudhary
235254600
A Semantic based Method for Unsupervised Commonsense Question Answering
Unsupervised commonsense question answering is appealing since it does not rely on any labeled task data. Among existing work, a popular solution is to use pre trained language models to score candidate choices directly conditioned on the question or context. However, such scores from language models can be easily affected by irrelevant factors, such as word frequencies, sentence structures, etc. These distracting factors may not only mislead the model to choose a wrong answer but also make it oversensitive to lexical perturbations in candidate answers. In this paper, we present a novel SEmanticbased Question Answering method (SEQA) for unsupervised commonsense question answering. Instead of directly scoring each answer choice, our method first generates a set of plausible answers with generative models (e.g. GPT 2) and then uses these plausible answers to select the correct choice by considering the semantic similarity between each plausible answer and each choice. We devise a simple, yet sound formalism for this idea and verify its effectiveness and robustness with extensive experiments. We evaluate the proposed method on four benchmark datasets, and our method achieves the best results in unsupervised settings. Moreover, when attacked by TextFooler (Jin et al. 2020) with synonym replacement, SEQA demonstrates much less performance drops than baselines, thereby indicating stronger robustness.
ACL/IJCNLP
Yilin Niu Fei Huang Jiaming Liang Wenkai Chen Xiaoyan Zhu Minlie Huang
234357557
Cross Modal Generative Augmentation for Visual Question Answering
Data augmentation is an approach that can effectively improve the performance of multimodal machine learning. This paper introduces a generative model for data augmentation by leveraging the correlations among multiple modalities. Different from conventional data augmentation approaches that apply low level operations with deterministic heuristics, our method proposes to learn an augmentation sampler that generates samples of the target modality conditioned on observed modalities in the variational auto encoder framework. Additionally, the proposed model is able to quantify the confidence of augmented data by its generative probability, and can be jointly updated with a downstream pipeline. Experiments on Visual Question Answering tasks demonstrate the effectiveness of the proposed generative model, which is able to boost the strong UpDn based models to the state of theart performance.
ArXiv
Zhen Wang Yishu Miao Lucia Specia
233296201
Generative Context Pair Selection for Multi hop Question Answering
Compositional reasoning tasks like multi hop question answering, require making latent decisions to get the final answer, given a question. However, crowdsourced datasets often capture only a slice of the underlying task distribution, which can induce unanticipated biases in models performing compositional reasoning. Furthermore, discriminatively trained models exploit such biases to get a better held out performance, without learning the right way to reason, as they do not necessitate paying attention to the question representation (conditioning variable) in its entirety, to estimate the answer likelihood. In this work, we propose a generative context selection model for multihop question answering that reasons about how the given question could have been generated given a context pair. While being comparable to the state of the art answering performance, our proposed generative passage selection model has a better performance (4.9% higher than baseline) on adversarial held out set which tests robustness of model's multihop reasoning capabilities.
ArXiv
Dheeru Dua Cicero Nogueira dos Santos Patrick Ng Ben Athiwaratkun Bing Xiang Matt Gardner Sameer Singh
233289483
Q2: Evaluating Factual Consistency in Knowledge Grounded Dialogues via Question Generation and Question Answering
Neural knowledge grounded generative models for dialogue often produce content that is factually inconsistent with the source text they rely on. As a consequence, such models are unreliable, limiting their real world applicability. Inspired by recent work on evaluating factual consistency in abstractive summarization (Durmus et al. 2020; Wang et al. 2020) we propose an automatic evaluation metric for factual consistency in knowledge grounded dialogue models using automatic question generation and question answering. Unlike previous works which use naive token based comparison of answer spans, our metric makes use of co reference resolution and natural language inference capabilities which greatly improve its performance. To foster proper evaluation, we curate a novel dataset of state of theart dialogue system outputs for the Wizard ofWikipedia dataset (Dinan et al. 2019) which we manually annotate for factual consistency. We perform a thorough meta evaluation of our metric against other metrics using the new dataset and two others, where it greatly outperforms the baselines.1
ArXiv
Or Honovich Leshem Choshen Roee Aharoni Ella Neeman Idan Szpektor Omri Abend
232478685
FeTaQA: Free form Table Question Answering
Existing table question answering datasets contain abundant factual questions that primarily evaluate the query and schema comprehension capability of a system, but they fail to include questions that require complex reasoning and integration of information due to the constraint of the associated short form answers. To address these issues and to demonstrate the full challenge of table question answering, we introduce FeTaQA, a new dataset with 10K Wikipediabased {table, question, free form answer, supporting table cells} pairs. FeTaQA yields a more challenging table question answering setting because it requires generating free form text answers after retrieval, inference, and integration of multiple discontinuous facts from a structured knowledge source. Unlike datasets of generative QA over text in which answers are prevalent with copies of short text spans from the source, answers in our dataset are human generated explanations involving entities and their high level relations. We provide two benchmark methods for the proposed task: a pipeline method based on semantic parsingbased QA systems and an end to end method based on large pretrained text generation models, and show that FeTaQA poses a challenge for both methods.
ArXiv
Linyong Nan Chia-Hsuan Hsieh Ziming Mao Neha Verma Rui Zhang Wojciech Kryscinski Nick Schoelkopf Riley Kong Xiangru Tang Murori Mutuma Benjamin Rosand Isabel Trindade Renusree Bandaru Jacob Cunningham Caiming Xiong Dragomir Radev
232185221
Conversational Answer Generation and Factuality for Reading Comprehension Question Answering
Question answering (QA) is an important use case on voice assistants. A popular approach to QA is extractive reading comprehension (RC) which finds an answer span in a text passage. However, extractive answers are often unnatural in a conversational context which results in suboptimal user experience. In this work, we investigate conversational answer generation for QA. We propose AnswerBART, an end to end generative RC model which combines answer generation from multiple passages with passage ranking and answerability. Moreover, a hurdle in applying generative RC are hallucinations where the answer is factually inconsistent with the passage text. We leverage recent work from summarization to evaluate factuality. Experiments show that AnswerBART significantly improves over previous best published results on MS MARCO 2.1 NLGEN by 2.5 ROUGE L and NarrativeQA by 9.4 ROUGE L.
ArXiv
Stanislav Peshterliev Barlas Oguz Debojeet Chatterjee Hakan Inan Vikas Bhardwaj
231662527
Zero shot Generalization in Dialog State Tracking through Generative Question Answering
Dialog State Tracking (DST) an integral part of modern dialog systems, aims to track user preferences and constraints (slots) in task oriented dialogs. In real world settings with constantly changing services, DST systems must generalize to new domains and unseen slot types. Existing methods for DST do not generalize well to new slot names and many require known ontologies of slot types and values for inference. We introduce a novel ontology free framework that supports natural language queries for unseen constraints and slots in multi domain task oriented dialogs. Our approach is based on generative question answering using a conditional language model pre trained on substantive English sentences. Our model improves joint goal accuracy in zero shot domain adaptation settings by up to 9% (absolute) over the previous state of the art on the MultiWOZ 2.1 dataset.
EACL
Shuyang Li Jin Cao Mukund Sridhar Henghui Zhu Shang-Wen Li Wael Hamza Julian McAuley
233132476
Research on Automatic Question Answering of Generative Knowledge Graph Based on Pointer Network
Question answering systems based on knowledge graphs are extremely challenging tasks in the field of natural language processing. Most of the existing Chinese Knowledge Base Question Answering(KBQA) can only return the knowledge stored in the knowledge base by extractive methods. Nevertheless, this processing does not conform to the reading habits and cannot solve the Out of vocabulary(OOV) problem. In this paper, a new generative question answering method based on knowledge graph is proposed, including three parts of knowledge vocabulary construction, data pre processing, and answer generation. In the word list construction, BiLSTM CRF is used to identify the entity in the source text, finding the triples contained in the entity, counting the word frequency, and constructing it. In the part of data pre processing, a pre trained language model BERT combining word frequency semantic features is adopted to obtain word vectors. In the answer generation part, one combination of a vocabulary constructed by the knowledge graph and a pointer generator network(PGN) is proposed to point to the corresponding entity for generating answer. The experimental results show that the proposed method can achieve superior performance on WebQA datasets than other methods.
Inf.
Shuang Liu Nannan Tan Yaqian Ge Niko Lukac