text
Understanding BARTScore 🛹
Understanding BARTScore 🛹
new_line
new_line
text
BARTScore is a text-generation evaluation metric that treats model evaluation as a text-generation task 🔄
BARTScore is a text-generation evaluation metric that treats model evaluation as a text-generation task 🔄
new_line
new_line
text
Other metrics approach the evaluation problem from different ML task perspectives; for instance, ROUGE and BLUE formulate it as an unsupervised matching task, BLUERT and COMET as a supervised regression, and BEER as a supervised ranking task.
Other metrics approach the evaluation problem from different ML task perspectives; for instance, ROUGE and BLUE formulate it as an unsupervised matching task, BLUERT and COMET as a supervised regression, and BEER as a supervised ranking task.
new_line
new_line
text
Meanwhile, BARTScore formulates it as a text-generation task. Its idea is to leverage BART's pre-trained contextual embeddings to return a score that measures either the faithfulness, precision, recall, or F-score response of the main text-generation model.
Meanwhile, BARTScore formulates it as a text-generation task. Its idea is to leverage BART's pre-trained contextual embeddings to return a score that measures either the faithfulness, precision, recall, or F-score response of the main text-generation model.
new_line
new_line
text
For example, if we want to measure faithfulness, the way it works is that we would take the source and the generated text from our model and use BART to calculate the log token probability of the generated text given the source; we can then weight those results and return the sum.
For example, if we want to measure faithfulness, the way it works is that we would take the source and the generated text from our model and use BART to calculate the log token probability of the generated text given the source; we can then weight those results and return the sum.
new_line
new_line
text
BARTScore correlates nicely with human scores, and it is relatively simple to implement.
BARTScore correlates nicely with human scores, and it is relatively simple to implement.
new_line
new_line
text
📑 Here is the original BARTScore paper: 
📑 Here is the original BARTScore paper: 
resource
https://huggingface.co/papers/2106.11520
paper
2106.11520
https://huggingface.co/papers/2106.11520
BARTScore: Evaluating Generated Text as Text Generation (2106.11520)
new_line
text
🧑‍💻 And the GitHub repo to use this metric: 
🧑‍💻 And the GitHub repo to use this metric: 
link
https://github.com/neulab/BARTScore
https://github.com/neulab/BARTScore
Understanding BARTScore 🛹

BARTScore is a text-generation evaluation metric that treats model evaluation as a text-generation task 🔄

Other metrics approach the evaluation problem from different ML task perspectives; for instance, ROUGE and BLUE formulate it as an unsupervised matching task, BLUERT and COMET as a supervised regression, and BEER as a supervised ranking task.

Meanwhile, BARTScore formulates it as a text-generation task. Its idea is to leverage BART's pre-trained contextual embeddings to return a score that measures either the faithfulness, precision, recall, or F-score response of the main text-generation model.

For example, if we want to measure faithfulness, the way it works is that we would take the source and the generated text from our model and use BART to calculate the log token probability of the generated text given the source; we can then weight those results and return the sum.

BARTScore correlates nicely with human scores, and it is relatively simple to implement.

📑 Here is the original BARTScore paper: https://huggingface.co/papers/2106.11520
🧑‍💻 And the GitHub repo to use this metric: https://github.com/neulab/BARTScore
👍
victor osanseviero clem dimpu01 ivanfioravanti firqaaa i0s YuvrajSingh9886