Mask More Than one Word:
image2226×813 85 KBHere, it says you can mask k tokens. However, in the documentation, it shows you only being able to mask one token. Is it possible to mask k words or am I mistaken?
2021-02-22T23:42:14Z
Are you using a fill-mask pipeline? If so, there’s a hard-coded limit of a single mask in the class, even though the model itself may support multiple masks. I guess the added behavior would warrant some more functionality when it comes to choosing how to sample, e.g. if there are N masked tokens with a few top-k probabilities each one might either want to sample from the join distribution (i.e. ranking the pairs based on p1*p2) or independently. The best approach would depend on the model’s internals, I suppose.I saw a post a while back welcoming a PR for this matter, so it’s a wanted feature.
2021-02-23T17:08:02Z
I was using the following. However, this code does not work well if you’re aiming to  consecutive tokens.import torch

sentence = "The capital of France <mask> contains the Eiffel <mask>."

token_ids = tokenizer.encode(sentence, return_tensors='pt')

# print(token_ids)

token_ids_tk = tokenizer.tokenize(sentence, return_tensors='pt')

print(token_ids_tk)

masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()

masked_pos = [mask.item() for mask in masked_position ]

print (masked_pos)

with torch.no_grad():

    output = model(token_ids)

last_hidden_state = output[0].squeeze()

print ("\n\n")

print ("sentence : ",sentence)

print ("\n")

list_of_list =[]

for mask_index in masked_pos:

    mask_hidden_state = last_hidden_state[mask_index]

    idx = torch.topk(mask_hidden_state, k=100, dim=0)[1]

    words = [tokenizer.decode(i.item()).strip() for i in idx]

    list_of_list.append(words)

    print (words)


best_guess = ""

for j in list_of_list:

    best_guess = best_guess+" "+j[0]---What I may try to do is <mask> two consecutive tokens. 

(ex: Paris is <mask> <mask> to visit.)

I'll then re-insert the most probable token for the first <mask>. 

(ex: Paris is a <mask> to visit.)

Then, I'll return the second <mask>'s most probable token. 

(ex: Paris is a city to visit.)
2021-02-23T17:34:19Z
This is the updated code.import torch

sentence = "The capital of France <mask> <mask> the Eiffel Tower."

token_ids = tokenizer.encode(sentence, return_tensors='pt')

token_ids_tk = tokenizer.tokenize(sentence, return_tensors='pt')

print(token_ids_tk)

masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()

masked_pos = [mask.item() for mask in masked_position ]

print (masked_pos)

with torch.no_grad():

    output = model(token_ids)

last_hidden_state = output[0].squeeze()

print ("\n\n")

print ("sentence : ",sentence)

print ("\n")

list_of_list =[]

for mask_index in masked_pos:

    mask_hidden_state = last_hidden_state[mask_index]

    idx = torch.topk(mask_hidden_state, k=5, dim=0)[1]

    words = [tokenizer.decode(i.item()).strip() for i in idx]

    list_of_list.append(words)

    #print(words)

sentences3 = []

for i in range(5):

    New = (list_of_list[0])

    New = New[i]

    New = New.replace("['", "").replace("']", "")

    sentence2 = sentence.replace("<mask>", New, 1)

    sentences3.append(sentence2)

for i in sentences3:

    #print(i)

    token_ids = tokenizer.encode(i, return_tensors='pt')

    token_ids_tk = tokenizer.tokenize(i, return_tensors='pt')

    #print(token_ids_tk)

    masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()

    masked_pos = [mask.item() for mask in masked_position ]

    print (masked_pos)

    with torch.no_grad():

        output = model(token_ids)

        last_hidden_state = output[0].squeeze()

        print ("\n\n")

        print ("sentence : ",i)

        print ("\n")

        list_of_list =[]

        for mask_index in masked_pos:

            mask_hidden_state = last_hidden_state[mask_index]

            idx = torch.topk(mask_hidden_state, k=5, dim=0)[1]

            words = [tokenizer.decode(i.item()).strip() for i in idx]

            list_of_list.append(words)

            print(words)-sentence :  The capital of France <mask> <mask> the Eiffel Tower.


[6]



sentence :  The capital of France , <mask> the Eiffel Tower.


['and', 'with', 'near', 'at', 'including']
[6]



sentence :  The capital of France is <mask> the Eiffel Tower.


['in', 'now', 'called', 'at', 'under']
[6]



sentence :  The capital of France lies <mask> the Eiffel Tower.


['atop', 'under', 'beneath', 'in', 'behind']
[6]



sentence :  The capital of France stands <mask> the Eiffel Tower.


['atop', 'at', 'on', 'under', 'in']
[6]



sentence :  The capital of France rests <mask> the Eiffel Tower.


['atop', 'on', 'in', 'upon', 'under']
2022-10-17T13:48:44Z
Hey, thanks for the code.Did you manage to solve it for consecutive masked tokens? I’m having the same problemCheers,Fran
2022-10-20T18:21:47Z
@franframI had the same issue… Do you happen to know which model supports multiple tokens during inference?
2022-10-21T14:33:11Z
@cnut1648I did, but it doesn’t work that well (I’m using a Spanish model, maybe an English one will work better).here’s a collab with the codehope it helpscheersGitHubGitHub - franfram/Multi-Mask-filling-model: Multi-mask filling modelMulti-mask filling model . Contribute to franfram/Multi-Mask-filling-model development by creating an account on GitHub.
2022-10-24T04:41:03Z
I see, thanks!!! I tried with English one and it seems that roberta for English also doesn’t work well…