
# SetFit

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
<!-- - **Sentence Transformer:** [Unknown](https://huggingface.co/unknown) -->
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 28 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.3768   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("jam1599/bert-text-classification-p5")
# Run inference
preds = model("love arising for your wife")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median  | Max |
|:-------------|:----|:--------|:----|
| Word count   | 1   | 28.2169 | 257 |

| Label | Training Sample Count |
|:------|:----------------------|
| loc5+ | 10                    |
| loc1  | 10                    |
| loc2  | 7                     |
| loc3  | 0                     |
| loc4  | 10                    |
| loc5  | 10                    |
| nfw   | 10                    |
| tfw   | 8                     |
| tfwc  | 2                     |
| tfwp  | 9                     |
| lay1  | 10                    |
| lay2  | 4                     |
| lay3  | 0                     |
| lay4  | 2                     |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (4, 4)
- max_steps: -1
- sampling_strategy: oversampling
- body_learning_rate: (2e-05, 1e-05)
- head_learning_rate: 0.01
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: True

### Training Results
| Epoch   | Step      | Training Loss | Validation Loss |
|:-------:|:---------:|:-------------:|:---------------:|
| 0.0003  | 1         | 0.1756        | -               |
| 0.0134  | 50        | 0.2671        | -               |
| 0.0268  | 100       | 0.2687        | -               |
| 0.0402  | 150       | 0.116         | -               |
| 0.0536  | 200       | 0.1965        | -               |
| 0.0671  | 250       | 0.316         | -               |
| 0.0805  | 300       | 0.2153        | -               |
| 0.0939  | 350       | 0.1917        | -               |
| 0.1073  | 400       | 0.1785        | -               |
| 0.1207  | 450       | 0.1722        | -               |
| 0.1341  | 500       | 0.1742        | -               |
| 0.1475  | 550       | 0.1889        | -               |
| 0.1609  | 600       | 0.1353        | -               |
| 0.1744  | 650       | 0.1128        | -               |
| 0.1878  | 700       | 0.1205        | -               |
| 0.2012  | 750       | 0.0717        | -               |
| 0.2146  | 800       | 0.081         | -               |
| 0.2280  | 850       | 0.0448        | -               |
| 0.2414  | 900       | 0.0589        | -               |
| 0.2548  | 950       | 0.1032        | -               |
| 0.2682  | 1000      | 0.0367        | -               |
| 0.2817  | 1050      | 0.0419        | -               |
| 0.2951  | 1100      | 0.0643        | -               |
| 0.3085  | 1150      | 0.0103        | -               |
| 0.3219  | 1200      | 0.0563        | -               |
| 0.3353  | 1250      | 0.0605        | -               |
| 0.3487  | 1300      | 0.1727        | -               |
| 0.3621  | 1350      | 0.0123        | -               |
| 0.3755  | 1400      | 0.0362        | -               |
| 0.3889  | 1450      | 0.0057        | -               |
| 0.4024  | 1500      | 0.0048        | -               |
| 0.4158  | 1550      | 0.0079        | -               |
| 0.4292  | 1600      | 0.0297        | -               |
| 0.4426  | 1650      | 0.0043        | -               |
| 0.4560  | 1700      | 0.0062        | -               |
| 0.4694  | 1750      | 0.0045        | -               |
| 0.4828  | 1800      | 0.0027        | -               |
| 0.4962  | 1850      | 0.0257        | -               |
| 0.5097  | 1900      | 0.0022        | -               |
| 0.5231  | 1950      | 0.002         | -               |
| 0.5365  | 2000      | 0.005         | -               |
| 0.5499  | 2050      | 0.0081        | -               |
| 0.5633  | 2100      | 0.008         | -               |
| 0.5767  | 2150      | 0.0062        | -               |
| 0.5901  | 2200      | 0.0012        | -               |
| 0.6035  | 2250      | 0.0021        | -               |
| 0.6170  | 2300      | 0.0016        | -               |
| 0.6304  | 2350      | 0.002         | -               |
| 0.6438  | 2400      | 0.002         | -               |
| 0.6572  | 2450      | 0.0018        | -               |
| 0.6706  | 2500      | 0.0016        | -               |
| 0.6840  | 2550      | 0.0029        | -               |
| 0.6974  | 2600      | 0.0049        | -               |
| 0.7108  | 2650      | 0.0016        | -               |
| 0.7242  | 2700      | 0.0009        | -               |
| 0.7377  | 2750      | 0.0016        | -               |
| 0.7511  | 2800      | 0.0009        | -               |
| 0.7645  | 2850      | 0.0009        | -               |
| 0.7779  | 2900      | 0.0012        | -               |
| 0.7913  | 2950      | 0.0005        | -               |
| 0.8047  | 3000      | 0.001         | -               |
| 0.8181  | 3050      | 0.0004        | -               |
| 0.8315  | 3100      | 0.0005        | -               |
| 0.8450  | 3150      | 0.0006        | -               |
| 0.8584  | 3200      | 0.0003        | -               |
| 0.8718  | 3250      | 0.0004        | -               |
| 0.8852  | 3300      | 0.0012        | -               |
| 0.8986  | 3350      | 0.0003        | -               |
| 0.9120  | 3400      | 0.0004        | -               |
| 0.9254  | 3450      | 0.0005        | -               |
| 0.9388  | 3500      | 0.0004        | -               |
| 0.9523  | 3550      | 0.0004        | -               |
| 0.9657  | 3600      | 0.0004        | -               |
| 0.9791  | 3650      | 0.0019        | -               |
| 0.9925  | 3700      | 0.0002        | -               |
| 1.0     | 3728      | -             | 0.309           |
| 1.0059  | 3750      | 0.0003        | -               |
| 1.0193  | 3800      | 0.0005        | -               |
| 1.0327  | 3850      | 0.0003        | -               |
| 1.0461  | 3900      | 0.0003        | -               |
| 1.0595  | 3950      | 0.0004        | -               |
| 1.0730  | 4000      | 0.0003        | -               |
| 1.0864  | 4050      | 0.0003        | -               |
| 1.0998  | 4100      | 0.0005        | -               |
| 1.1132  | 4150      | 0.0005        | -               |
| 1.1266  | 4200      | 0.0002        | -               |
| 1.1400  | 4250      | 0.0006        | -               |
| 1.1534  | 4300      | 0.0002        | -               |
| 1.1668  | 4350      | 0.0002        | -               |
| 1.1803  | 4400      | 0.0004        | -               |
| 1.1937  | 4450      | 0.0002        | -               |
| 1.2071  | 4500      | 0.0003        | -               |
| 1.2205  | 4550      | 0.0003        | -               |
| 1.2339  | 4600      | 0.0003        | -               |
| 1.2473  | 4650      | 0.0002        | -               |
| 1.2607  | 4700      | 0.0002        | -               |
| 1.2741  | 4750      | 0.0001        | -               |
| 1.2876  | 4800      | 0.0002        | -               |
| 1.3010  | 4850      | 0.0002        | -               |
| 1.3144  | 4900      | 0.0002        | -               |
| 1.3278  | 4950      | 0.0001        | -               |
| 1.3412  | 5000      | 0.0004        | -               |
| 1.3546  | 5050      | 0.0003        | -               |
| 1.3680  | 5100      | 0.0002        | -               |
| 1.3814  | 5150      | 0.0003        | -               |
| 1.3948  | 5200      | 0.0003        | -               |
| 1.4083  | 5250      | 0.0002        | -               |
| 1.4217  | 5300      | 0.0002        | -               |
| 1.4351  | 5350      | 0.0003        | -               |
| 1.4485  | 5400      | 0.0002        | -               |
| 1.4619  | 5450      | 0.0003        | -               |
| 1.4753  | 5500      | 0.0001        | -               |
| 1.4887  | 5550      | 0.0002        | -               |
| 1.5021  | 5600      | 0.0004        | -               |
| 1.5156  | 5650      | 0.0003        | -               |
| 1.5290  | 5700      | 0.0002        | -               |
| 1.5424  | 5750      | 0.0001        | -               |
| 1.5558  | 5800      | 0.0002        | -               |
| 1.5692  | 5850      | 0.0003        | -               |
| 1.5826  | 5900      | 0.0002        | -               |
| 1.5960  | 5950      | 0.0004        | -               |
| 1.6094  | 6000      | 0.0009        | -               |
| 1.6229  | 6050      | 0.0002        | -               |
| 1.6363  | 6100      | 0.0001        | -               |
| 1.6497  | 6150      | 0.0004        | -               |
| 1.6631  | 6200      | 0.0002        | -               |
| 1.6765  | 6250      | 0.0002        | -               |
| 1.6899  | 6300      | 0.0002        | -               |
| 1.7033  | 6350      | 0.0002        | -               |
| 1.7167  | 6400      | 0.0003        | -               |
| 1.7302  | 6450      | 0.0002        | -               |
| 1.7436  | 6500      | 0.0002        | -               |
| 1.7570  | 6550      | 0.0003        | -               |
| 1.7704  | 6600      | 0.0001        | -               |
| 1.7838  | 6650      | 0.0001        | -               |
| 1.7972  | 6700      | 0.0004        | -               |
| 1.8106  | 6750      | 0.0002        | -               |
| 1.8240  | 6800      | 0.0001        | -               |
| 1.8374  | 6850      | 0.0001        | -               |
| 1.8509  | 6900      | 0.0001        | -               |
| 1.8643  | 6950      | 0.0001        | -               |
| 1.8777  | 7000      | 0.0001        | -               |
| 1.8911  | 7050      | 0.0002        | -               |
| 1.9045  | 7100      | 0.0001        | -               |
| 1.9179  | 7150      | 0.0002        | -               |
| 1.9313  | 7200      | 0.0002        | -               |
| 1.9447  | 7250      | 0.0001        | -               |
| 1.9582  | 7300      | 0.0002        | -               |
| 1.9716  | 7350      | 0.0002        | -               |
| 1.9850  | 7400      | 0.0002        | -               |
| 1.9984  | 7450      | 0.0007        | -               |
| 2.0     | 7456      | -             | 0.3158          |
| 2.0118  | 7500      | 0.0003        | -               |
| 2.0252  | 7550      | 0.0002        | -               |
| 2.0386  | 7600      | 0.0001        | -               |
| 2.0520  | 7650      | 0.0002        | -               |
| 2.0655  | 7700      | 0.0002        | -               |
| 2.0789  | 7750      | 0.0002        | -               |
| 2.0923  | 7800      | 0.0004        | -               |
| 2.1057  | 7850      | 0.0002        | -               |
| 2.1191  | 7900      | 0.0003        | -               |
| 2.1325  | 7950      | 0.0001        | -               |
| 2.1459  | 8000      | 0.0001        | -               |
| 2.1593  | 8050      | 0.0001        | -               |
| 2.1727  | 8100      | 0.0004        | -               |
| 2.1862  | 8150      | 0.0002        | -               |
| 2.1996  | 8200      | 0.0002        | -               |
| 2.2130  | 8250      | 0.0002        | -               |
| 2.2264  | 8300      | 0.0002        | -               |
| 2.2398  | 8350      | 0.0001        | -               |
| 2.2532  | 8400      | 0.0002        | -               |
| 2.2666  | 8450      | 0.0001        | -               |
| 2.2800  | 8500      | 0.0002        | -               |
| 2.2935  | 8550      | 0.0002        | -               |
| 2.3069  | 8600      | 0.0001        | -               |
| 2.3203  | 8650      | 0.0001        | -               |
| 2.3337  | 8700      | 0.0001        | -               |
| 2.3471  | 8750      | 0.0001        | -               |
| 2.3605  | 8800      | 0.0005        | -               |
| 2.3739  | 8850      | 0.0002        | -               |
| 2.3873  | 8900      | 0.0001        | -               |
| 2.4008  | 8950      | 0.0002        | -               |
| 2.4142  | 9000      | 0.0001        | -               |
| 2.4276  | 9050      | 0.0001        | -               |
| 2.4410  | 9100      | 0.0001        | -               |
| 2.4544  | 9150      | 0.0002        | -               |
| 2.4678  | 9200      | 0.0001        | -               |
| 2.4812  | 9250      | 0.0001        | -               |
| 2.4946  | 9300      | 0.0003        | -               |
| 2.5080  | 9350      | 0.0003        | -               |
| 2.5215  | 9400      | 0.0001        | -               |
| 2.5349  | 9450      | 0.0001        | -               |
| 2.5483  | 9500      | 0.0002        | -               |
| 2.5617  | 9550      | 0.0001        | -               |
| 2.5751  | 9600      | 0.0001        | -               |
| 2.5885  | 9650      | 0.0002        | -               |
| 2.6019  | 9700      | 0.0001        | -               |
| 2.6153  | 9750      | 0.0001        | -               |
| 2.6288  | 9800      | 0.0001        | -               |
| 2.6422  | 9850      | 0.0001        | -               |
| 2.6556  | 9900      | 0.0           | -               |
| 2.6690  | 9950      | 0.0001        | -               |
| 2.6824  | 10000     | 0.0002        | -               |
| 2.6958  | 10050     | 0.0001        | -               |
| 2.7092  | 10100     | 0.0002        | -               |
| 2.7226  | 10150     | 0.0002        | -               |
| 2.7361  | 10200     | 0.0001        | -               |
| 2.7495  | 10250     | 0.0001        | -               |
| 2.7629  | 10300     | 0.0001        | -               |
| 2.7763  | 10350     | 0.0001        | -               |
| 2.7897  | 10400     | 0.0           | -               |
| 2.8031  | 10450     | 0.0001        | -               |
| 2.8165  | 10500     | 0.0001        | -               |
| 2.8299  | 10550     | 0.0001        | -               |
| 2.8433  | 10600     | 0.0001        | -               |
| 2.8568  | 10650     | 0.0001        | -               |
| 2.8702  | 10700     | 0.0002        | -               |
| 2.8836  | 10750     | 0.0002        | -               |
| 2.8970  | 10800     | 0.0001        | -               |
| 2.9104  | 10850     | 0.0002        | -               |
| 2.9238  | 10900     | 0.0001        | -               |
| 2.9372  | 10950     | 0.0001        | -               |
| 2.9506  | 11000     | 0.0002        | -               |
| 2.9641  | 11050     | 0.0001        | -               |
| 2.9775  | 11100     | 0.0001        | -               |
| 2.9909  | 11150     | 0.0001        | -               |
| **3.0** | **11184** | **-**         | **0.307**       |
| 3.0043  | 11200     | 0.0001        | -               |
| 3.0177  | 11250     | 0.0002        | -               |
| 3.0311  | 11300     | 0.0001        | -               |
| 3.0445  | 11350     | 0.0001        | -               |
| 3.0579  | 11400     | 0.0           | -               |
| 3.0714  | 11450     | 0.0001        | -               |
| 3.0848  | 11500     | 0.0001        | -               |
| 3.0982  | 11550     | 0.0001        | -               |
| 3.1116  | 11600     | 0.0001        | -               |
| 3.125   | 11650     | 0.0001        | -               |
| 3.1384  | 11700     | 0.0001        | -               |
| 3.1518  | 11750     | 0.0           | -               |
| 3.1652  | 11800     | 0.0001        | -               |
| 3.1786  | 11850     | 0.0           | -               |
| 3.1921  | 11900     | 0.0001        | -               |
| 3.2055  | 11950     | 0.0002        | -               |
| 3.2189  | 12000     | 0.0001        | -               |
| 3.2323  | 12050     | 0.0001        | -               |
| 3.2457  | 12100     | 0.0001        | -               |
| 3.2591  | 12150     | 0.0001        | -               |
| 3.2725  | 12200     | 0.0001        | -               |
| 3.2859  | 12250     | 0.0001        | -               |
| 3.2994  | 12300     | 0.0001        | -               |
| 3.3128  | 12350     | 0.0001        | -               |
| 3.3262  | 12400     | 0.0001        | -               |
| 3.3396  | 12450     | 0.0001        | -               |
| 3.3530  | 12500     | 0.0002        | -               |
| 3.3664  | 12550     | 0.0001        | -               |
| 3.3798  | 12600     | 0.0001        | -               |
| 3.3932  | 12650     | 0.0           | -               |
| 3.4067  | 12700     | 0.0001        | -               |
| 3.4201  | 12750     | 0.0001        | -               |
| 3.4335  | 12800     | 0.0001        | -               |
| 3.4469  | 12850     | 0.0001        | -               |
| 3.4603  | 12900     | 0.0001        | -               |
| 3.4737  | 12950     | 0.0001        | -               |
| 3.4871  | 13000     | 0.0001        | -               |
| 3.5005  | 13050     | 0.0           | -               |
| 3.5139  | 13100     | 0.0001        | -               |
| 3.5274  | 13150     | 0.0           | -               |
| 3.5408  | 13200     | 0.0001        | -               |
| 3.5542  | 13250     | 0.0001        | -               |
| 3.5676  | 13300     | 0.0           | -               |
| 3.5810  | 13350     | 0.0001        | -               |
| 3.5944  | 13400     | 0.0001        | -               |
| 3.6078  | 13450     | 0.0001        | -               |
| 3.6212  | 13500     | 0.0001        | -               |
| 3.6347  | 13550     | 0.0001        | -               |
| 3.6481  | 13600     | 0.0001        | -               |
| 3.6615  | 13650     | 0.0001        | -               |
| 3.6749  | 13700     | 0.0001        | -               |
| 3.6883  | 13750     | 0.0001        | -               |
| 3.7017  | 13800     | 0.0001        | -               |
| 3.7151  | 13850     | 0.0           | -               |
| 3.7285  | 13900     | 0.0001        | -               |
| 3.7420  | 13950     | 0.0001        | -               |
| 3.7554  | 14000     | 0.0001        | -               |
| 3.7688  | 14050     | 0.0           | -               |
| 3.7822  | 14100     | 0.0001        | -               |
| 3.7956  | 14150     | 0.0001        | -               |
| 3.8090  | 14200     | 0.0001        | -               |
| 3.8224  | 14250     | 0.0001        | -               |
| 3.8358  | 14300     | 0.0001        | -               |
| 3.8492  | 14350     | 0.0           | -               |
| 3.8627  | 14400     | 0.0001        | -               |
| 3.8761  | 14450     | 0.0           | -               |
| 3.8895  | 14500     | 0.0001        | -               |
| 3.9029  | 14550     | 0.0001        | -               |
| 3.9163  | 14600     | 0.0001        | -               |
| 3.9297  | 14650     | 0.0           | -               |
| 3.9431  | 14700     | 0.0001        | -               |
| 3.9565  | 14750     | 0.0           | -               |
| 3.9700  | 14800     | 0.0001        | -               |
| 3.9834  | 14850     | 0.0           | -               |
| 3.9968  | 14900     | 0.0001        | -               |
| 4.0     | 14912     | -             | 0.31            |

* The bold row denotes the saved checkpoint.
### Framework Versions
- Python: 3.10.12
- SetFit: 1.0.2
- Sentence Transformers: 2.2.2
- Transformers: 4.35.2
- PyTorch: 2.1.0+cu121
- Datasets: 2.16.1
- Tokenizers: 0.15.0

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->
{"library_name": "setfit", "tags": ["setfit", "sentence-transformers", "text-classification", "generated_from_setfit_trainer"], "metrics": ["accuracy"], "widget": [{"text": "and so you could potentially do a little biasing that way"}, {"text": "its there and then i feel a sense of fullness really"}, {"text": "love arising for your wife"}, {"text": "but at the same time it had some of these nondual qualities"}, {"text": "layer two that thats spaciousness of that distance"}], "pipeline_tag": "text-classification", "inference": true, "model-index": [{"name": "SetFit", "results": [{"task": {"type": "text-classification", "name": "Text Classification"}, "dataset": {"name": "Unknown", "type": "unknown", "split": "test"}, "metrics": [{"type": "accuracy", "value": 0.3767605633802817, "name": "Accuracy"}]}]}]}
setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 model-index region:us