
# LEVER (for Codex on Spider)

This is one of the models produced by the paper ["LEVER: Learning to Verify Language-to-Code Generation with Execution"](https://arxiv.org/abs/2302.08468).

**Authors:** [Ansong Ni](https://niansong1996.github.io), Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang*, Xi Victoria Lin*

**Note**: This specific model is for Codex on the [Spider](https://yale-lily.github.io/spider) dataset, for the models pretrained on other datasets, please see:
* [lever-wikitq-codex](https://huggingface.co/niansong1996/lever-wikitq-codex)
* [lever-gsm8k-codex](https://huggingface.co/niansong1996/lever-gsm8k-codex)
* [lever-mbpp-codex](https://huggingface.co/niansong1996/lever-mbpp-codex)

![Model Image](https://huggingface.co/niansong1996/lever-spider-codex/resolve/main/lever_info.png)

# Model Details


## Model Description
The advent of pre-trained code language models (Code LLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine CodeLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics cannot well capture the semantic features of the execution results, such as data type and value range, which often indicates the correctness of the program. In this work, we propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the CodeLM is correct or not based on the natural language input, the program itself and its execution results. The sampled programs are reranked by combining the verification score with the CodeLM generation probability, and marginalizing over programs with the same execution results. On four datasets across the domains of table QA, math QA and basic Python programming, LEVER consistently improves over the base CodeLMs (4.6% to 10.9% with code-davinci-002) and achieves new state-of-the-art results on all of them.


- **Developed by:** Yale University and Meta AI
- **Shared by [Optional]:** Ansong Ni

- **Model type:** Text Classification
- **Language(s) (NLP):** More information needed
- **License:** Apache-2.0
- **Parent Model:** T5-base
- **Resources for more information:**
 	- [Github Repo](https://github.com/niansong1996/lever)
 	- [Associated Paper](https://arxiv.org/abs/2302.08468)


# Uses
 

## Direct Use

This model is *not* intended to be directly used. LEVER is used to verify and rerank the programs generated by code LLMs (e.g., Codex). We recommend checking out our [Github Repo](https://github.com/niansong1996/lever) for more details.
 
## Downstream Use

LEVER is learned to verify and rerank the programs sampled from code LLMs for different tasks. 
More specifically, for `lever-spider-codex`, it was trained on the outputs of `code-davinci-002` on the [Spider](https://yale-lily.github.io/spider) dataset. It can be used to rerank the SQL programs generated by Codex out-of-box.
Moreover, it may also be applied to other model's outputs on the Spider dataset, as studied in the [Original Paper](https://arxiv.org/abs/2302.08468).


 
## Out-of-Scope Use
 
The model should not be used to intentionally create hostile or alienating environments for people. 
 
# Bias, Risks, and Limitations
 
 
Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.



## Recommendations
 
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

# Training Details
 
## Training Data
 
The model is trained with the outputs from `code-davinci-002` model on the [Spider](https://yale-lily.github.io/spider) dataset.
 
## Training Procedure

20 program samples are drawn from the Codex model on the training examples of the Spider dataset, those programs are later executed to obtain the execution information.
And for each example and its program sample, the natural language description and execution information are also part of the inputs that used to train the T5-based model to predict "yes" or "no" as the verification labels.

 
### Preprocessing
 
Please follow the instructions in the [Github Repo](https://github.com/niansong1996/lever) to reproduce the results.


 
### Speeds, Sizes, Times
 
More information needed 


 
# Evaluation
 
 
## Testing Data, Factors & Metrics
 
### Testing Data
 
Dev set of the [Spider](https://yale-lily.github.io/spider) dataset.
 
### Factors
More information needed
 
### Metrics
 
Execution accuracy (i.e., pass@1)
 
 
## Results 
 

### Spider Text-to-SQL Generation

|                 | Exec. Acc. |
|-----------------|------------|
| Codex           |     75.3   |
| Codex+LEVER     |     81.9   |



 
# Model Examination

More information needed

 
# Environmental Impact
 
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
- **Hardware Type:** More information needed
- **Hours used:** More information needed
- **Cloud Provider:** More information needed
- **Compute Region:** More information needed
- **Carbon Emitted:** More information needed
 
# Technical Specifications [optional]
 
## Model Architecture and Objective
 
`lever-spider-codex` is based on T5-base.

## Compute Infrastructure
 
More information needed
 
### Hardware
 
 
More information needed
 
### Software
 
More information needed.
 
# Citation

 
**BibTeX:**
 
 
```bibtex
@inproceedings{ni2023lever,
  title={Lever: Learning to verify language-to-code generation with execution},
  author={Ni, Ansong and Iyer, Srini and Radev, Dragomir and Stoyanov, Ves and Yih, Wen-tau and Wang, Sida I and Lin, Xi Victoria},
  booktitle={Proceedings of the 40th International Conference on Machine Learning (ICML'23)},
  year={2023}
}
```
 
 
 
 
# Glossary [optional]
 
More information needed

# More Information [optional]
More information needed 

 
# Model Card Author and Contact
 
Ansong Ni, contact info on [personal website](https://niansong1996.github.io)
 
# How to Get Started with the Model
 
This model is *not* intended to be directly used, please follow the instructions in the [Github Repo](https://github.com/niansong1996/lever).
{"license": "apache-2.0", "datasets": ["spider"], "metrics": ["accuracy"], "model-index": [{"name": "lever-spider-codex", "results": [{"task": {"type": "code generation"}, "dataset": {"name": "spider (text-to-sql)", "type": "spider"}, "metrics": [{"type": "accuracy", "value": 81.9, "verified": false}]}]}]}
transformers pytorch t5 text2text-generation dataset:spider arxiv:2302.08468 arxiv:1910.09700 license:apache-2.0 model-index autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #t5 #text2text-generation #dataset-spider #arxiv-2302.08468 #arxiv-1910.09700 #license-apache-2.0 #model-index #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

LEVER (for Codex on Spider)
===========================


This is one of the models produced by the paper "LEVER: Learning to Verify Language-to-Code Generation with Execution".


Authors: Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang\*, Xi Victoria Lin\*


Note: This specific model is for Codex on the Spider dataset, for the models pretrained on other datasets, please see:


* lever-wikitq-codex
* lever-gsm8k-codex
* lever-mbpp-codex


!Model Image


Model Details
=============


Model Description
-----------------


The advent of pre-trained code language models (Code LLMs) has led to significant progress in language-to-code generation. State-of-the-art approaches in this area combine CodeLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics cannot well capture the semantic features of the execution results, such as data type and value range, which often indicates the correctness of the program. In this work, we propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the CodeLM is correct or not based on the natural language input, the program itself and its execution results. The sampled programs are reranked by combining the verification score with the CodeLM generation probability, and marginalizing over programs with the same execution results. On four datasets across the domains of table QA, math QA and basic Python programming, LEVER consistently improves over the base CodeLMs (4.6% to 10.9% with code-davinci-002) and achieves new state-of-the-art results on all of them.


* Developed by: Yale University and Meta AI
* Shared by [Optional]: Ansong Ni
* Model type: Text Classification
* Language(s) (NLP): More information needed
* License: Apache-2.0
* Parent Model: T5-base
* Resources for more information:


	+ Github Repo
	+ Associated Paper


Uses
====


Direct Use
----------


This model is *not* intended to be directly used. LEVER is used to verify and rerank the programs generated by code LLMs (e.g., Codex). We recommend checking out our Github Repo for more details.


Downstream Use
--------------


LEVER is learned to verify and rerank the programs sampled from code LLMs for different tasks.
More specifically, for 'lever-spider-codex', it was trained on the outputs of 'code-davinci-002' on the Spider dataset. It can be used to rerank the SQL programs generated by Codex out-of-box.
Moreover, it may also be applied to other model's outputs on the Spider dataset, as studied in the Original Paper.


Out-of-Scope Use
----------------


The model should not be used to intentionally create hostile or alienating environments for people.


Bias, Risks, and Limitations
============================


Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.


Recommendations
---------------


Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.


Training Details
================


Training Data
-------------


The model is trained with the outputs from 'code-davinci-002' model on the Spider dataset.


Training Procedure
------------------


20 program samples are drawn from the Codex model on the training examples of the Spider dataset, those programs are later executed to obtain the execution information.
And for each example and its program sample, the natural language description and execution information are also part of the inputs that used to train the T5-based model to predict "yes" or "no" as the verification labels.


### Preprocessing


Please follow the instructions in the Github Repo to reproduce the results.


### Speeds, Sizes, Times


More information needed


Evaluation
==========


Testing Data, Factors & Metrics
-------------------------------


### Testing Data


Dev set of the Spider dataset.


### Factors


More information needed


### Metrics


Execution accuracy (i.e., pass@1)


Results
-------


### Spider Text-to-SQL Generation



Model Examination
=================


More information needed


Environmental Impact
====================


Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).


* Hardware Type: More information needed
* Hours used: More information needed
* Cloud Provider: More information needed
* Compute Region: More information needed
* Carbon Emitted: More information needed


Technical Specifications [optional]
===================================


Model Architecture and Objective
--------------------------------


'lever-spider-codex' is based on T5-base.


Compute Infrastructure
----------------------


More information needed


### Hardware


More information needed


### Software


More information needed.


BibTeX:


Glossary [optional]
===================


More information needed


More Information [optional]
===========================


More information needed


Model Card Author and Contact
=============================


Ansong Ni, contact info on personal website


How to Get Started with the Model
=================================


This model is *not* intended to be directly used, please follow the instructions in the Github Repo.



### Preprocessing


Please follow the instructions in the Github Repo to reproduce the results. ### Speeds, Sizes, Times


More information needed


Evaluation
==========


Testing Data, Factors & Metrics
------------------------------- ### Testing Data


Dev set of the Spider dataset. ### Factors


More information needed ### Metrics


Execution accuracy (i.e., pass@1)


Results
------- ### Spider Text-to-SQL Generation



Model Examination
=================


More information needed


Environmental Impact
====================


Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).


* Hardware Type: More information needed
* Hours used: More information needed
* Cloud Provider: More information needed
* Compute Region: More information needed
* Carbon Emitted: More information needed


Technical Specifications [optional]
===================================


Model Architecture and Objective
--------------------------------


'lever-spider-codex' is based on T5-base.


Compute Infrastructure
----------------------


More information needed ### Hardware


More information needed ### Software


More information needed.


BibTeX:


Glossary [optional]
===================


More information needed


More Information [optional]
===========================


More information needed


Model Card Author and Contact
=============================


Ansong Ni, contact info on personal website


How to Get Started with the Model
=================================


This model is *not* intended to be directly used, please follow the instructions in the Github Repo.
TAGS
#transformers #pytorch #t5 #text2text-generation #dataset-spider #arxiv-2302.08468 #arxiv-1910.09700 #license-apache-2.0 #model-index #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
 ### Preprocessing


Please follow the instructions in the Github Repo to reproduce the results. ### Speeds, Sizes, Times


More information needed


Evaluation
==========


Testing Data, Factors & Metrics
------------------------------- ### Testing Data


Dev set of the Spider dataset. ### Factors


More information needed ### Metrics


Execution accuracy (i.e., pass@1)


Results
------- ### Spider Text-to-SQL Generation



Model Examination
=================


More information needed


Environmental Impact
====================


Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).


* Hardware Type: More information needed
* Hours used: More information needed
* Cloud Provider: More information needed
* Compute Region: More information needed
* Carbon Emitted: More information needed


Technical Specifications [optional]
===================================


Model Architecture and Objective
--------------------------------


'lever-spider-codex' is based on T5-base.


Compute Infrastructure
----------------------


More information needed ### Hardware


More information needed ### Software


More information needed.


BibTeX:


Glossary [optional]
===================


More information needed


More Information [optional]
===========================


More information needed


Model Card Author and Contact
=============================


Ansong Ni, contact info on personal website


How to Get Started with the Model
=================================


This model is *not* intended to be directly used, please follow the instructions in the Github Repo.
TAGS
#transformers #pytorch #t5 #text2text-generation #dataset-spider #arxiv-2302.08468 #arxiv-1910.09700 #license-apache-2.0 #model-index #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
### Preprocessing


Please follow the instructions in the Github Repo to reproduce the results.### Speeds, Sizes, Times


More information needed


Evaluation
==========


Testing Data, Factors & Metrics
-------------------------------### Testing Data


Dev set of the Spider dataset.### Factors


More information needed### Metrics


Execution accuracy (i.e., pass@1)


Results
-------### Spider Text-to-SQL Generation



Model Examination
=================


More information needed


Environmental Impact
====================


Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).


* Hardware Type: More information needed
* Hours used: More information needed
* Cloud Provider: More information needed
* Compute Region: More information needed
* Carbon Emitted: More information needed


Technical Specifications [optional]
===================================


Model Architecture and Objective
--------------------------------


'lever-spider-codex' is based on T5-base.


Compute Infrastructure
----------------------


More information needed### Hardware


More information needed### Software


More information needed.


BibTeX:


Glossary [optional]
===================


More information needed


More Information [optional]
===========================


More information needed


Model Card Author and Contact
=============================


Ansong Ni, contact info on personal website


How to Get Started with the Model
=================================


This model is *not* intended to be directly used, please follow the instructions in the Github Repo.