Connecting to hub account from Databricks
I’m running transformers in a Databricks notebook with a local dataset. the task is text classification with BERT and DistilBERT. I have no problem loading public checkpoints from the hub and fine-tuning. The problem comes when I want to push the model back to my account on the hub. The cellfrom huggingface_hub import notebook_loginnotebook_login()doesn’t allow me to log in. Figuring I need to display HTML I trieddisplayHTML(notebook_login())but this gives me a java NullPointerException. Has anybody succeeded in connecting to their account from a DataBricks notebook?thanks
2022-06-18T21:52:19Z
Hi Alun!Today I ran into the same challenge. Hugging Face unfortunately seems to lack proper documentation on how to login into the Hugging Face Hub from within a Databricks notebook. While I was unsuccessful in logging in through the currently documentedhuggingface-cli login,notebook_login()andHfApi.set_access_token(), I was successful in logging into and pushing models and datasets to the Hugging Face Hub through a hacky implementation of what is supposed to be or become a deprecated method.Start with installing Hugging Face Hub through:%pip install huggingface_hubAnd installing git-lfs through:%sh
# https://stackoverflow.com/questions/48734119/git-lfs-is-not-a-git-command-unclear
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfsThen, continue with logging in through:from huggingface_hub.commands.user import _login
from huggingface_hub import HfApi

api = HfApi()
_login(REDACTED_TOKEN = api, username = "USER", password = "PASS")And:%sh
git config --global credential.helper store
# https://github.com/huggingface/notebooks/blob/main/examples/language_modeling_from_scratch-tf.ipynb
git config --global user.email "EMAIL"
git config --global user.name "FULLNAME"This worked for me and took quite some trial and error to figure out. Pushing models still behaves a bit weird due to some occasionally occuring git errors.I hope this helps!
2022-06-24T17:00:51Z
It seems_login()has seen been updated. You can still follow the login procedure above for the most part, but you need to use a token instead of username/password combination with _login():from huggingface_hub.commands.user import _login
from huggingface_hub import HfApi

api = HfApi()
_login(REDACTED_TOKEN = api, token = "TOKEN")Tokens can be generated in theAccess Tokenspage of your Hugging Face profile.
2022-06-28T08:26:46Z
Thanks. This looks reasonable, but I haven’t been able to get it to work as yet. I removed huggingface_hub from Databricks and reinstalled it (from Pypi). Looking at the code on GitHub, I can see what you are doing, and token is a keyword argument (although REDACTED_TOKEN isn’t)._login(api, token="TOKEN")However, I’m getting the same problem. I checked when Pypi was updated last, and it appears to be after the last change to github. Just to be clear, I am using my actual token there, not the string “TOKEN”. My guess is that Databricks didn’t actually upgrade the library.
2023-05-10T09:17:15Z
I go through the code but It was not work for me, I think it is regarding new updates, but you can use directly _login from _login.pyfrom huggingface_hub._login import _login
_login(token='your token as string', add_to_git_credential=False)