Include more features per token while training the BERT model
My data is in this format for representing journey of one user per row.Item 1, Item 2, Item 3, Item 7, Item 10Item Description 1, Item Description 2…Item Descripiton 10Item 1 purchased on Monday, Item 2 purchased on Tuesday, …Item 10 purchased on Tuesday…Item 1 Price, Item 2 Price…Item 10 PriceConsider for 10 users, you shall have 10 such rows of journey which has Item list, Item Description list, Item Purchased day list, Item price list.Currently, we have trained MLM on data of all users with only item list using BertForMaskedLM from huggingface transformers libraryJourney is transformed to: Item 1, Item 2, Item 3, [MASK], Item 10And then in inferencing we try to predict the next item the user should select by providing his history to the model.Now, we want to include description, price, day of purchase etc as more features. These features are per token and not per user.Need help to understand the architecture and code changes for custom BERT library.