
# longformer_4096_qsi

This model is a fine-tuned version of [mrm8488/longformer-base-4096-finetuned-squadv2](https://huggingface.co/mrm8488/longformer-base-4096-finetuned-squadv2) on a tiny [NovelQSI](https://huggingface.co/datasets/Kkordik/NovelQSI) dataset.
It achieves the following results on the evaluation set:
- Loss: 2.9598

## Model description

This model is a test model for my research project. The idea of the model is to understand which novel character said the requested quote. 
It achieves a bit better results on the ´test´ split of the NovelQSI dataset than base longformer-base-4096-finetuned-squadv2 model on the same dataset split.

**Base model results:**

```
{
  "exact_match": {
    "confidence_interval": [8.754452551305853, 14.718614718614718],
    "score": 12.121212121212121,
    "standard_error": 1.8579217243778676
  },
  "f1": {
    "confidence_interval": [18.469101076147584, 28.28409063313956],
    "score": 22.799422799422796,
    "standard_error": 2.896728175757627
  },
  "latency_in_seconds": 0.7730605573419919,
  "samples_per_second": 1.2935597224598967,
  "total_time_in_seconds": 178.5769887460001
}

```

 **Achieved results:**
 
```
{
  "exact_match": {
    "confidence_interval": [16.017316017316016, 24.242424242424242],
    "score": 20.346320346320347,
    "standard_error": 2.9434375492784994
  },
  "f1": {
    "confidence_interval": [23.123469058324783, 31.823648733317036],
    "score": 26.580086580086572,
    "standard_error": 2.593030474995015
  },
  "latency_in_seconds": 0.8093855569913422,
  "samples_per_second": 1.235505120349827,
  "total_time_in_seconds": 186.96806366500005
}
```

The results have shown, that the technique has its future.

## Training and evaluation data

You can find training code in the github repo of my research:

https://github.com/Kkordik/NovelQSI

It was trained and evaluated in notebooks, so it is easy to reproduce.

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| No log        | 1.0   | 93   | 3.0886          |
| No log        | 1.99  | 186  | 3.3755          |
| No log        | 2.99  | 279  | 2.9598          |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.0+cu118
- Datasets 2.15.0
- Tokenizers 0.15.0
{"language": ["en"], "license": "apache-2.0", "tags": ["generated_from_trainer"], "datasets": ["Kkordik/NovelQSI"], "base_model": "mrm8488/longformer-base-4096-finetuned-squadv2", "widget": [{"text": "Which character said 'You know, I read somewhere that the brightest stars are those that have undergone the most turmoil. Maybe it's the same with us \u2013 our struggles make us shine brighter'?", "context": "Characters:\nAlex: Aliases: {'Alex'}. Gender: Male, The character is: major Bella: Aliases: {'Bella'}. Gender: Female, The character is: major Charlie: Aliases: {'Charlie'}. Gender: Non-binary, The character is: major\nSummary: \nIn the novel's previous section, Alex, Bella, and Charlie, three friends in Luminara, engage in a deep conversation at The Starry Night caf\u00e9. They debate destiny, with Alex believing in fate, Bella advocating for self-made destiny, and Charlie suggesting a combination of both. Personal reflections emerge, such as Bella's musings on heartbreak and Alex's thoughts on longing. Charlie compares people's struggles to stars, implying that challenges enhance personal growth. The night progresses with their varied, meaningful discussions.\nNovel Text:\nIn the bustling city of Luminara, three friends, Alex, Bella, and Charlie, often met at their favorite caf\u00e9, The Starry Night, to discuss life, love, and the mysteries of the universe. The caf\u00e9, with its warm ambiance and the soft hum of jazz in the background, provided the perfect setting for their deep conversations.\nOne evening, as the city lights twinkled outside, the trio found themselves engrossed in a discussion about destiny.\nAlex, a firm believer in fate, argued passionately, \"I truly believe that our paths are predestined. The universe has a plan for each of us, and all our choices lead us to our ultimate destiny.\"\nBella, a skeptic, laughed softly and countered, \"That's a romantic notion, Alex, but I think we make our own destiny. It's our decisions, not some cosmic plan, that shape our lives.\"\nCharlie, always the mediator, added thoughtfully, \"Maybe it's a bit of both. Perhaps there's a grand design, but within it, we have the freedom to make choices that influence our journey.\"\nTheir conversation drifted to other topics as the evening wore on. At one point, Bella, reflecting on a recent heartbreak, said, \"Sometimes, I wonder if the heart ever truly heals from loss, or if it just learns to live with the pain.\"\nAlex, looking out the window at the starry sky, mused, \"It's strange how the heart yearns for what it can't have. The unattainable always seems so much more alluring.\"\nCharlie, who had been quiet for a while, suddenly spoke up with a gleam in his eye, \"You know, I read somewhere that the brightest stars are those that have undergone the most turmoil. Maybe it's the same with us \u2013 our struggles make us shine brighter.\"\nAs the night deepened, their conversation meandered through various topics."}], "model-index": [{"name": "Kkordik/test_longformer_4096_qsi", "results": [{"task": {"type": "question-answering"}, "dataset": {"name": "NovelQSI", "type": "Kkordik/NovelQSI", "split": "test"}, "metrics": [{"type": "exact_match", "value": 20.346, "verified": false}, {"type": "f1", "value": 26.58, "verified": false}]}]}]}
transformers tensorboard safetensors longformer question-answering generated_from_trainer en dataset:Kkordik/NovelQSI base_model:mrm8488/longformer-base-4096-finetuned-squadv2 license:apache-2.0 model-index endpoints_compatible region:us