Take a look at the following script to get a better idea of how to run ZeRO Inference without the [Trainer] on a model that won't fit on a single GPU.

!/usr/bin/env python
This script demonstrates how to use Deepspeed ZeRO in an inference mode when one can't fit a model
into a single GPU

1. Use 1 GPU with CPU offload
2. Or use multiple GPUs instead

First you need to install deepspeed: pip install deepspeed