
# INT8 GPT-J 6B

GPT-J 6B is a transformer model trained using Ben Wang's [Mesh Transformer JAX](https://github.com/kingoflolz/mesh-transformer-jax/). "GPT-J" refers to the class of model, while "6B" represents the number of trainable parameters.

This repository contains GPT-J 6B onnx model suitable for building TensorRT int8+fp32 engines. Quantization of model was performed by the [ENOT-AutoDL](https://pypi.org/project/enot-autodl/) framework. Code for building of TensorRT engines and examples published on [github](https://github.com/ENOT-AutoDL/ENOT-transformers).

## Metrics:

|   |TensorRT INT8+FP32|torch FP16|torch FP32|
|---|:---:|:---:|:---:|
| **Lambada Acc** |78.46%|79.53%|-|
| **Model size (GB)**  |8.5|12.1|24.2|

### Test environment

* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116

## Latency:

|Input sequance length|Number of generated tokens|TensorRT INT8+FP32 ms|torch FP16 ms|Acceleration|
|:---:|:---:|:---:|:---:|:---:|
|64|64|1040|1610|1.55|
|64|128|2089|3224|1.54|
|64|256|4236|6479|1.53|
|128|64|1060|1619|1.53|
|128|128|2120|3241|1.53|
|128|256|4296|6510|1.52|
|256|64|1109|1640|1.49|
|256|128|2204|3276|1.49|
|256|256|4443|6571|1.49|

### Test environment

* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116

## How to use

Example of inference and accuracy test [published on github](https://github.com/ENOT-AutoDL/ENOT-transformers):
```shell
git clone https://github.com/ENOT-AutoDL/ENOT-transformers
```

{"language": ["en"], "license": "apache-2.0", "library_name": "transformers", "tags": ["text-generation-inference", "causal-lm", "int8", "tensorrt", "ENOT-AutoDL"], "datasets": ["lambada"], "pipeline_tag": "text-generation"}
transformers onnx text-generation-inference causal-lm int8 tensorrt ENOT-AutoDL text-generation en dataset:lambada license:apache-2.0 endpoints_compatible region:us
TAGS
#transformers #onnx #text-generation-inference #causal-lm #int8 #tensorrt #ENOT-AutoDL #text-generation #en #dataset-lambada #license-apache-2.0 #endpoints_compatible #region-us 

INT8 GPT-J 6B
=============


GPT-J 6B is a transformer model trained using Ben Wang's Mesh Transformer JAX. "GPT-J" refers to the class of model, while "6B" represents the number of trainable parameters.


This repository contains GPT-J 6B onnx model suitable for building TensorRT int8+fp32 engines. Quantization of model was performed by the ENOT-AutoDL framework. Code for building of TensorRT engines and examples published on github.


Metrics:
--------



### Test environment


* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116


Latency:
--------



### Test environment


* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116


How to use
----------


Example of inference and accuracy test published on github:



### Test environment


* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116


Latency:
-------- ### Test environment


* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116


How to use
----------


Example of inference and accuracy test published on github:
TAGS
#transformers #onnx #text-generation-inference #causal-lm #int8 #tensorrt #ENOT-AutoDL #text-generation #en #dataset-lambada #license-apache-2.0 #endpoints_compatible #region-us 
 ### Test environment


* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116


Latency:
-------- ### Test environment


* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116


How to use
----------


Example of inference and accuracy test published on github:
TAGS
#transformers #onnx #text-generation-inference #causal-lm #int8 #tensorrt #ENOT-AutoDL #text-generation #en #dataset-lambada #license-apache-2.0 #endpoints_compatible #region-us 
### Test environment


* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116


Latency:
--------### Test environment


* GPU RTX 4090
* CPU 11th Gen Intel(R) Core(TM) i7-11700K
* TensorRT 8.5.3.1
* pytorch 1.13.1+cu116


How to use
----------


Example of inference and accuracy test published on github: