`serving` signature in TensorFlow Serving blogpost
Hi everyone!I am currently working through@jplu'sblogposton serving a HuggingFace model with TF-Serving, in which he overwrites the model’sservingmethod to change the signature of the traced graph input to accept embeddings.That, in turn, led me to discover that thisservingsignature is part of all TF models (Models — transformers 4.7.0 documentation)Can someone explain to me how exactly thisservingmethod is used by the model server? I can’t find it referenced in the rest of the tutorial and I wasn’t succesful in finding my way around the codebase.Is that redefined signature used at all in the tutorial? I might be mistaken, but it seems to me that the requests (both REST and gRPC) to the TF-server use the output of the tokenizer, not those of an embedding layer.
2021-08-05T20:03:19Z
cc@Rocketknight1
2021-08-09T08:17:55Z
I missed the fact that theservingmethod is explicitly exported as a metagraph in the SavedModel for all TF models (see here)EDIT:One thing that’s still not completely clear to me are the redefined inputs. The tutorial changes theservingmethod to accept token embeddings, yet the requests to the model use the output of the tokenizer, so token ids. What am I not seeing?