How do I reduce DistilBERT model size?
Hi everyone,I am recently start using huggingface’s transformer library and used BERT model to fit my data, after training on AWS sagemaker exported model is 300+ MB each. Then I tried distilBERT, it reduced to around 200MB, yet still too big to invoke if put into multi model endpoint. Is there anyway to reduce the size of distilBERT even more so I can fit them in the multi model endpoint?
2021-04-01T17:46:04Z
Hi ! what makes it too big to invoke in multi-model endpoint? not enough time to deploy? invocation too slow?
2021-04-02T08:01:57Z
Check Do we really need to distill? Jont Loss is all we need.GitHubGitHub - legacyai/tf-transformers: State of the art faster Transformer with...State of the art faster Transformer with Tensorflow 2.0 ( NLP, Computer Vision, Audio ). - GitHub - legacyai/tf-transformers: State of the art faster Transformer with Tensorflow 2.0 ( NLP, Computer...This model is 4x smaller than DistillBERT and better than it by GLUE score of 4 points.
2021-04-06T00:45:05Z
It seems that after I put BERT models in a multi-model model and create an endpoint for it, when I tried to invoke the endpoint by targeting specific model, it will timeout. And BERT models are stored in S3 buckets. So I assume invocation is too slow.
2021-04-06T21:13:26Z
hey@burrtthere’s a few threads on the forum about making transformer models smaller / faster, e.g.Deploy multilingual sentence tansformer into cloudDeploying inference model size and performancemy standard recommendation is to try quantization followed by ONNX / ONNX Runtime. before doing that though, i’d first try to understand what’s causing the timeout on your endpoint - it might be unrelated to the model and you don’t want to spend a lot of time optimising the wrong thing
2021-04-07T18:44:11Z
I agree with@lewtuni’d first diagnose what’s causing the timeout. You can add logs to your inference functions (model loading and inference) and check theModelLoadingWaitTimeSageMaker endpoint metric in CloudWatch to check what steps take time. If model loading is the issue,according to the SageMaker docit may speed things up if you use adtype instance, that have local NVMe SSDs instead of using virtual EBS drives for storage. Also, try using bigger instances (in AWS, network bandwidth is generally proportional to instance size).If using regular SageMaker endpoints is enough (and not SageMaker MME), you can take a look at that demo that deploys the biggerbert-base-casedon SageMakerGitHub - aws-samples/amazon-sagemaker-bert-classify-pytorch: This sample show you how to train BERT on Amazon Sagemaker using Spot instances
2021-04-12T21:11:41Z
Thank you for your reply. SageMaker engpoints(not MME) is enough. After I deploy the endpoint with one single BERT model, it responds normally and can predict. After I deploy a multi-model endpoint with one or more BERT models in it, the cloudwatch just says model_fn is not provided yet it’s in the script that is in the model artifacts.