
## Model Details: INT8 GPT-J 6B

GPT-J 6B is a transformer model trained using Ben Wang's [Mesh Transformer JAX](https://github.com/kingoflolz/mesh-transformer-jax/). "GPT-J" refers to the class of model, while "6B" represents the number of trainable parameters.

This int8 ONNX model is generated by [neural-compressor](https://github.com/intel/neural-compressor) and the fp32 model can be exported with below command:
```shell
python -m transformers.onnx --model=EleutherAI/gpt-j-6B onnx_gptj/ --framework pt --opset 13 --feature=causal-lm-with-past
```

| Model Detail | Description |
| ----------- | ----------- | 
| Model Authors - Company | Intel | 
| Date | April 10, 2022 | 
| Version | 1 | 
| Type | Text Generation | 
| Paper or Other Resources | - | 
| License | Apache 2.0 |
| Questions or Comments | [Community Tab](https://huggingface.co/Intel/gpt-j-6B-int8-dynamic/discussions)|

| Intended Use | Description |
| ----------- | ----------- | 
| Primary intended uses | You can use the raw model for text generation inference | 
| Primary intended users | Anyone doing text generation inference | 
| Out-of-scope uses | This model in most cases will need to be fine-tuned for your particular task.  The model should not be used to intentionally create hostile or alienating environments for people.|


### How to use

Download the model and script by cloning the repository:
```shell
git clone https://huggingface.co/Intel/gpt-j-6B-int8-dynamic
```

Then you can do inference based on the model and script 'evaluation.ipynb'.

## Metrics (Model Performance):
| Model  | Model Size (GB) | Lambada Acc |
|---|:---:|:---:|
| FP32 |23|0.7954|
| INT8 |6|0.7926|


{"language": ["en"], "license": "apache-2.0", "library_name": "transformers", "tags": ["text-generation-inference", "causal-lm", "int8", "ONNX", "PostTrainingDynamic", "Intel\u00ae Neural Compressor", "neural-compressor"], "datasets": ["lambada"], "pipeline_tag": "text-generation"}
transformers onnx gptj text-generation text-generation-inference causal-lm int8 ONNX PostTrainingDynamic Intel® Neural Compressor neural-compressor en dataset:lambada license:apache-2.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #onnx #gptj #text-generation #text-generation-inference #causal-lm #int8 #ONNX #PostTrainingDynamic #Intel® Neural Compressor #neural-compressor #en #dataset-lambada #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 

Model Details: INT8 GPT-J 6B
----------------------------


GPT-J 6B is a transformer model trained using Ben Wang's Mesh Transformer JAX. "GPT-J" refers to the class of model, while "6B" represents the number of trainable parameters.


This int8 ONNX model is generated by neural-compressor and the fp32 model can be exported with below command:




### How to use


Download the model and script by cloning the repository:


Then you can do inference based on the model and script 'URL'.


Metrics (Model Performance):
----------------------------




### How to use


Download the model and script by cloning the repository:


Then you can do inference based on the model and script 'URL'.


Metrics (Model Performance):
----------------------------
TAGS
#transformers #onnx #gptj #text-generation #text-generation-inference #causal-lm #int8 #ONNX #PostTrainingDynamic #Intel® Neural Compressor #neural-compressor #en #dataset-lambada #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
 ### How to use


Download the model and script by cloning the repository:


Then you can do inference based on the model and script 'URL'.


Metrics (Model Performance):
----------------------------
TAGS
#transformers #onnx #gptj #text-generation #text-generation-inference #causal-lm #int8 #ONNX #PostTrainingDynamic #Intel® Neural Compressor #neural-compressor #en #dataset-lambada #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
### How to use


Download the model and script by cloning the repository:


Then you can do inference based on the model and script 'URL'.


Metrics (Model Performance):
----------------------------