
# SetFit with sentence-transformers/paraphrase-mpnet-base-v2

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 5 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|:------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| LOC5+ | <ul><li>'hard to follow what people are saying'</li><li>'former sense of self that fell away'</li><li>'okay so um so you know its you know at least location six um then you you describe things like feeling like its just the universe or existence thats looking out of out of the eyes um or experiencing itself so thats you know something that people can say at location seven um but its definitely a very accurate description of location eight um can be nine'</li></ul>                                                                                                                                                                           |
| LOC1  | <ul><li>'physiological ailments most likely remain as well'</li><li>'um most people dont really retain fluidity to location one but it does happen'</li><li>'emotional or energetic almost feels like an entity'</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| LOC3  | <ul><li>'but it becomes more salient and'</li><li>'if youve just got some patterns that unwind as a function of deconditioning and what returns whether there is a joy some sort of intense joy you know underneath all of that or whatnot'</li><li>'celestial standpoint'</li></ul>                                                                                                                                                                                                                                                                                                                                                                           |
| LOC4  | <ul><li>'unusual perceptual effects'</li><li>'not being able to connect properly to my family'</li><li>'right so much so that like both of our coats smelled like it for a while after a little bit right and and we were and it led to this discussion about frustration and location for right because that experience was absolutely experienced as you know frustration for her and location for at that moment'</li></ul>                                                                                                                                                                                                                                 |
| LOC2  | <ul><li>'when it comes to feelings or sensations ive never been good at recognizing how as a system was ever actually feeling so its a very neutral if theres a feeling to it its extremely neutral because ive looked to see whether you know love or those active more active alive kind of qualities to me this seems just very quiet'</li><li>'so many people will get to like layer two or something of location two and theyll think oh'</li><li>'if you went in the direction of layer three of location two um youd sort of avoid those and really sink into the center looseness stillness silence really dissolution of any sense of self'</li></ul> |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.4494   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("dendimaki/apeiron-v1")
# Run inference
preds = model("single emotional expression")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median | Max |
|:-------------|:----|:-------|:----|
| Word count   | 1   | 23.85  | 231 |

| Label | Training Sample Count |
|:------|:----------------------|
| LOC1  | 8                     |
| LOC2  | 8                     |
| LOC3  | 8                     |
| LOC4  | 8                     |
| LOC5+ | 8                     |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (4, 4)
- max_steps: -1
- sampling_strategy: oversampling
- body_learning_rate: (2e-05, 1e-05)
- head_learning_rate: 0.01
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: True

### Training Results
| Epoch   | Step    | Training Loss | Validation Loss |
|:-------:|:-------:|:-------------:|:---------------:|
| 0.0125  | 1       | 0.2498        | -               |
| 0.625   | 50      | 0.1882        | -               |
| 1.0     | 80      | -             | 0.2764          |
| 1.25    | 100     | 0.0261        | -               |
| 1.875   | 150     | 0.0044        | -               |
| 2.0     | 160     | -             | 0.2693          |
| 2.5     | 200     | 0.0018        | -               |
| 3.0     | 240     | -             | 0.2696          |
| 3.125   | 250     | 0.0024        | -               |
| 3.75    | 300     | 0.0022        | -               |
| **4.0** | **320** | **-**         | **0.2688**      |

* The bold row denotes the saved checkpoint.
### Framework Versions
- Python: 3.10.12
- SetFit: 1.0.3
- Sentence Transformers: 2.2.2
- Transformers: 4.35.2
- PyTorch: 2.1.0+cu121
- Datasets: 2.16.1
- Tokenizers: 0.15.0

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->
{"library_name": "setfit", "tags": ["setfit", "sentence-transformers", "text-classification", "generated_from_setfit_trainer"], "metrics": ["accuracy"], "widget": [{"text": "so if we go from there and we go on to like another location location six six is notorious for its glitches"}, {"text": "single emotional expression"}, {"text": "increasing aliveness and power"}, {"text": "there is no listener or seer"}, {"text": "disassembly and reintegration"}], "pipeline_tag": "text-classification", "inference": true, "base_model": "sentence-transformers/paraphrase-mpnet-base-v2", "model-index": [{"name": "SetFit with sentence-transformers/paraphrase-mpnet-base-v2", "results": [{"task": {"type": "text-classification", "name": "Text Classification"}, "dataset": {"name": "Unknown", "type": "unknown", "split": "test"}, "metrics": [{"type": "accuracy", "value": 0.449438202247191, "name": "Accuracy"}]}]}]}
setfit safetensors mpnet sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:sentence-transformers/paraphrase-mpnet-base-v2 model-index region:us