Using the Tokenizers library in a Unity project
Hi everyone! Any advice on the best way to leverage the Tokenizers library when using a transformer model in a Unity project (C#)? I’m considering creating a Rust plugin, embedding Python in a C++ plugin, or using a server. Ideally, I could run everything locally, model size allowing.