Custom inference endpoint with multiple models
I would like to deploy an inference endpoint based on a combination of multiple models.Here is the repository structure/handler.py
/requirements.txt
/model1.pt
/model2.ptWhat will be the value ofpathhere? Adding a few prints in the handler tells me that it is set to/repository/.However, it also looks like/repository/does not containmodel1.ptandmodel2.pt.Is this on purpose? How can I update the code below to accessmodel1.ptandmodel2.pt?class EndpointHandler:
    def __init__(self, path=""):
        self.model1 = torch.load('model1.pt')
        self.model2 = torch.load('model2.pt')

    def __call__(self, data):
        pass
2024-01-12T08:40:13Z
the torch load should be `os.path.join(path,“model1.pt”). The path var provided in the init points to where your files are.
2024-01-12T12:27:10Z
Thanks.The problem was that, for some reason, not all files are pulled from the model repository.For instance, files with.ckptextension (it was the case for me) are not available inpathdirectory. I had to change the extension of the model files for them to be downloaded.This behavior is probably documented somewhere but I could not find it.The problem is now solved.os.path.join(path, "model.pt")does the trick.
2024-01-13T00:27:35Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.