How to configure GPU server-side batching with SageMaker HF Hosting?
How to configure GPU server-side batching with SageMaker HF Hosting?I want to be able to process multiple GPU inferences in one single batched call (vs doing CPU-GPU travel at every inference). MMS supports that in its open-source flavor, wondering how we can use it with SM HF?
2022-05-04T11:29:13Z
Not sure how the MMS side of things works maybe you can ask atIssues · awslabs/multi-model-server · GitHub.But once MMS is configured it depends on the task you are using it might be possible that you need to create some custom logic using ainference.pyand overwriting theinput_fnoroutpuf_fn.