collaborativeearth/bge-m3-aicacia

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("driante/bge-m3_finetuned")
# Run inference
sentences = [
    'abandoned fishponds carbon storage',
    "We also examined the changes in carbon stocks from the landward to seaward zones. We hypothesized that since it has been 30 years after the abandoned fishpond rehabilitation, the reforested site must have already fully recovered from this past disturbance. However, we presumed that the aboveand-belowground carbon at the reforestation site could not exceed that of the natural stand, given the altered soil's physical and chemical properties upon fishpond establishment. Our study supports the broader call for abandoned aquaculture rehabilitation and protection of mangrove forests from further destruction to sustain their roles in mitigating the impacts of climate change. We especially highlight the carbon sequestration and storage potentials of mangrove-reverted abandoned fishponds by generating a pool of reliable data and information for sound mangrove forest management interventions.",
    '#### 2.1 A Phased, Risk-Based Approach for FIs to Reach Full Alignment over Time\n\nRecognizing that FIs today have different levels of\n\ncapacity to implement Paris alignment activities, we\n\npropose a phased approach whereby DFIs would require\n\nFIs to meet certain requirements in phase 1, when the\n\ncontract between the DFI and FI is being approved, and\n\nadditional requirements in phase 2. This approach aims\n\nto balance the need for robust criteria that minimize\n\nthe risk of misaligned investments with that of finding a\n\nworkable solution for capacity-constrained FIs.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.5109
cosine_accuracy@3	0.6901
cosine_accuracy@5	0.7568
cosine_accuracy@10	0.8267
cosine_precision@1	0.5109
cosine_precision@3	0.23
cosine_precision@5	0.1514
cosine_precision@10	0.0827
cosine_recall@1	0.5109
cosine_recall@3	0.6901
cosine_recall@5	0.7568
cosine_recall@10	0.8267
cosine_ndcg@10	0.6669
cosine_mrr@10	0.6159
cosine_map@100	0.6219
dot_accuracy@1	0.5109
dot_accuracy@3	0.6901
dot_accuracy@5	0.7568
dot_accuracy@10	0.8267
dot_precision@1	0.5109
dot_precision@3	0.23
dot_precision@5	0.1514
dot_precision@10	0.0827
dot_recall@1	0.5109
dot_recall@3	0.6901
dot_recall@5	0.7568
dot_recall@10	0.8267
dot_ndcg@10	0.6669
dot_mrr@10	0.6159
dot_map@100	0.6219

Training Details

Training Dataset

Unnamed Dataset

• Size: 9,908 training samples
• Columns: queries and text
• Approximate statistics based on the first 1000 samples:

  	queries	text
  type	string	string
  details	min: 3 tokens mean: 11.06 tokens max: 60 tokens	min: 4 tokens mean: 308.67 tokens max: 761 tokens

• Samples:

  queries	text
  what appendix is a potential reference	###### APPENDIX 4: OTHER POTENTIAL REFERENCES AND TOOLS
  capacity building needs toward low carbon	## ▪ [Capacity building needs toward low-carbon ] industry growth.
  which planting is best for carbon sequestration	Carter et al., 2008). Geometry may also be important; for instance, narrow linear plantings often have greater biomass than block plantings due to higher resource availability at their edges (Paul et al., 2013a). At a given site and topographic location, the diversity of plants, typically measured as species richness, has been shown to also influence above ground biomass (Hooper et al., 2005). However, most research supporting this relationship has been conducted in grasslands and microbial microcosms with fewer investigations of woody species assemblages in field settings (Cardinale et al., 2012). A recent meta-analysis, of the few published experimental studies of woody species growth, indicated that species richness had a positive effect on aboveground biomass, but noted that the identity and functional traits of the species involved are an important component of how much carbon will likely be sequestered (Hulvey et al., 2013). Furthermore, individual studies can show conflicting results (e.g. Potvin et al., 2011) suggesting context dependency is important. In addition to plant species number and identity, the density of established individuals potentially influences the amount of carbon sequestered by an environmental planting (Dwyer et al., 2010a;Paul et al., 2013a). Surveys of recently established non-experimental environmental plantings (and old growth forest e.g. Jacob et al., 2010;Seidel et al., 2013), as compared to experimental approaches, provide an alternative avenue to investigate potential influences on biomass/productivity, especially if 1) the planting is large enough to capture spatial heterogeneity (e.g. > 100 ha); 2) there are records of establishment practices and site conditions; and, 3) records include accurate identities and measurements of established individuals. The first two allow investigations of biomass relationships among and within soil -vegetation type associations while the latter provides important information so that suitable allometric relationships can be applied to estimate biomass and carbon (e.g. Jonson and Freudenberger, 2011).

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• learning_rate: 1e-05
• num_train_epochs: 4
• warmup_ratio: 0.1
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	cosine_map@100
0	0	-	0.5914
0.0807	100	0.1772	0.6148
0.1614	200	0.1152	0.6267
0.2421	300	0.111	0.6339
0.3228	400	0.0723	0.6327
0.4036	500	0.0817	0.6311
0.4843	600	0.0644	0.6234
0.5650	700	0.1018	0.6330
0.6457	800	0.0689	0.6318
0.7264	900	0.0701	0.6290
0.8071	1000	0.0766	0.6254
0.8878	1100	0.0837	0.6383
0.9685	1200	0.0853	0.6279
1.0492	1300	0.0572	0.6354
1.1299	1400	0.0397	0.6340
1.2107	1500	0.0342	0.6384
1.2914	1600	0.0307	0.6351
1.3721	1700	0.0149	0.6313
1.4528	1800	0.013	0.6258
1.5335	1900	0.0175	0.6313
1.6142	2000	0.027	0.6383
1.6949	2100	0.0138	0.6331
1.7756	2200	0.0164	0.6298
1.8563	2300	0.0167	0.6333
1.9370	2400	0.0298	0.6328
2.0178	2500	0.0144	0.6253
2.0985	2600	0.0073	0.6308
2.1792	2700	0.007	0.6283
2.2599	2800	0.0088	0.6313
2.3406	2900	0.0043	0.6281
2.4213	3000	0.0081	0.6256
2.5020	3100	0.0041	0.6238
2.5827	3200	0.0053	0.6271
2.6634	3300	0.0062	0.6252
2.7441	3400	0.0061	0.6242
2.8249	3500	0.006	0.6247
2.9056	3600	0.0064	0.6277
2.9863	3700	0.0072	0.6229
3.0670	3800	0.0028	0.6239
3.1477	3900	0.0031	0.6225
3.2284	4000	0.0038	0.6217
3.3091	4100	0.0025	0.6200
3.3898	4200	0.0048	0.6230
3.4705	4300	0.0015	0.6211
3.5513	4400	0.0023	0.6207
3.6320	4500	0.0045	0.6235
3.7127	4600	0.0046	0.6227
3.7934	4700	0.0019	0.6225
3.8741	4800	0.0038	0.6223
3.9548	4900	0.0035	0.6219

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.1.1
• Transformers: 4.45.2
• PyTorch: 2.5.1+cu121
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}