How to solve factual inconsistency when fine tuning
I fine tuned the pszemraj/led-large-book-summary model on financial reports and was able to get a rouge-1 score close to 0.6But after manually checking eachmodel generated summarywith the actual financial report, it turned out that mostkey information specially numbers are wrong.And i compared the summary got from openai GPT4 using same financial report and all the numbers and key information was correct in it.I would like to know your experiences, workarounds and ideas on this issue and how to handle factual inconsistency.Thanks
2024-10-01T14:49:59Z
Seems like you are asian . I don’t know anything about hugging face I wanted a help . I am from Nepal and I wanted to use hugging face but the problem is I am confused with hugging face billing .  Pls help me :My dollar card is provided by my local bank, where my bank account is created. Even though it is prepaid it is under the visa category and I guess Hugging Face accepts visa cards. Even though that card is not linked with my bank account directly. I understand hugging face has banned prepaid cards due to fraud issues but my dollar card is legal and is only provided by my bank unless we have account in that particular bank in order to create an account there are strict policies so there is no way a scammer will be able to create that particular dollar card ? Will hugging face accept my card? I guess the billing card system is managed by Stripe and even though stripe is not available in my country I hope I can pay through my Dollar Card. Please help me I beg you…
2024-10-01T14:51:04Z
Which card are you using could you please tell me and which country?
2024-10-01T15:10:24Z
Im using huggingface models locally so no need to pay anything since they are free.
2024-10-01T15:36:05Z
Oh so how much limit is for hugging face free tier / local?
2024-10-01T15:37:16Z
I want to create code generator type of tool from user prompt . Can it be possible in free tier ? What are the limitations actually I am new in hugging face could you help me with it .
2024-10-01T15:57:57Z
The “factual” inconsistency that you have is tough to enforce to the model when fine tuning it.According to me, such pattern might come from the architecture itself which is related to LongFormer. LongFormer uses sliding window and “global” representation of text when computing the attention as explained in the paper :pszemraj/led-large-book-summaryThis kind of architecture may compress a lot of token level information into a more global one.For your task, I would recommend using a small LLM or any model in the category text summarization that relies on the “vanilla” transformers architecture and already pretrained and/or fine tuned on financial datasets that are more likely to contain numbers !
2024-10-02T06:21:35Z
no limit if you are download and run the models, datasets in your device.  And there is free spaces to run your application in huggingface.You need to pay only if you are hosting models and use GPU to run them.huggingface.coIntroduction - Hugging Face NLP CourseWe’re on a journey to advance and democratize artificial intelligence through open source and open science.this course will give a brief intro to huggingface.
2024-10-02T14:57:39Z
Thank you. I had tried out other models like T5,  Pegasus, llama and mistrals.  They are factually accurate than the LED models specially with numbers. But after fine tuning  these models summary rouge scores were less than the LED’s. And the LED had the best human like summary with a 16k context size.But as you said will try out smaller models with 1B, 1.5B parameters.
2024-10-02T15:14:41Z
hey !Maybe give a try to this one : human-centered-summarization/financial-summarization-pegasusMost of the financial data is numbers linked so it may be worth !
2024-10-03T07:09:14Z
Thank you againAnd after referring that model found this another fine-tuned version of that model.huggingface.coNerdward/financial-summarization-pegasus-finetuned-pytorch-model · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.will try out this also.