modernbert-embed-base-bible

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: nomic-ai/modernbert-embed-base
  • Maximum Sequence Length: 8192 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity
  • Training Dataset:
    • json
  • Language: fr
  • License: apache-2.0

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Steve77/modernbert-embed-base-bible")
# Run inference
sentences = [
    "Quelles tâches les Lévites devaient-ils accomplir dans le service de la maison de l'Éternel?",
    "Ils devaient prendre soin des parvis et des chambres, purifier toutes les choses saintes, s'occuper des pains de proposition, de la fleur de farine pour les offrandes, des galettes sans levain, des gâteaux cuits sur la plaque et des gâteaux frits, et de toutes les mesures de capacité et de longueur.",
    "Les chefs des maisons paternelles, les chefs des tribus d'Israël, les chefs de milliers et de centaines, et les intendants du roi.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Metric dim_768 dim_512 dim_256 dim_128 dim_64
cosine_accuracy@1 0.175 0.1716 0.1602 0.1437 0.1107
cosine_accuracy@3 0.2484 0.2402 0.2276 0.2047 0.1595
cosine_accuracy@5 0.2762 0.272 0.2603 0.2331 0.1862
cosine_accuracy@10 0.3203 0.3162 0.3075 0.2794 0.2272
cosine_precision@1 0.175 0.1716 0.1602 0.1437 0.1107
cosine_precision@3 0.0828 0.0801 0.0759 0.0682 0.0532
cosine_precision@5 0.0552 0.0544 0.0521 0.0466 0.0372
cosine_precision@10 0.032 0.0316 0.0308 0.0279 0.0227
cosine_recall@1 0.175 0.1716 0.1602 0.1437 0.1107
cosine_recall@3 0.2484 0.2402 0.2276 0.2047 0.1595
cosine_recall@5 0.2762 0.272 0.2603 0.2331 0.1862
cosine_recall@10 0.3203 0.3162 0.3075 0.2794 0.2272
cosine_ndcg@10 0.2443 0.2395 0.2284 0.2057 0.1633
cosine_ndcg@15 0.2525 0.2464 0.2357 0.2141 0.17
cosine_ndcg@20 0.2574 0.2517 0.2405 0.2194 0.1746
cosine_mrr@10 0.2205 0.2155 0.2038 0.1829 0.1435
cosine_map@100 0.2276 0.2226 0.2108 0.1901 0.15

Training Details

Training Dataset

json

  • Dataset: json
  • Size: 47,560 training samples
  • Columns: anchor and positive
  • Approximate statistics based on the first 1000 samples:
    anchor positive
    type string string
    details
    • min: 8 tokens
    • mean: 21.11 tokens
    • max: 45 tokens
    • min: 3 tokens
    • mean: 24.84 tokens
    • max: 108 tokens
  • Samples:
    anchor positive
    Quels sont les noms des fils de Schobal? Aljan, Manahath, Ébal, Schephi et Onam
    Quels sont les noms des fils de Tsibeon? Ajja et Ana
    Qui est le fils d'Ana? Dischon
  • Loss: MatryoshkaLoss with these parameters:
    {
        "loss": "MultipleNegativesRankingLoss",
        "matryoshka_dims": [
            768,
            512,
            256,
            128,
            64
        ],
        "matryoshka_weights": [
            1,
            1,
            1,
            1,
            1
        ],
        "n_dims_per_step": -1
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • gradient_accumulation_steps: 16
  • learning_rate: 2e-05
  • num_train_epochs: 4
  • lr_scheduler_type: cosine
  • warmup_ratio: 0.1
  • bf16: True
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 16
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 4
  • max_steps: -1
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: True
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss dim_768_cosine_ndcg@20 dim_512_cosine_ndcg@20 dim_256_cosine_ndcg@20 dim_128_cosine_ndcg@20 dim_64_cosine_ndcg@20
0.0538 10 12.274 - - - - -
0.1076 20 11.5084 - - - - -
0.1615 30 10.5276 - - - - -
0.2153 40 9.0432 - - - - -
0.2691 50 7.572 - - - - -
0.3229 60 7.7696 - - - - -
0.3767 70 6.5673 - - - - -
0.4305 80 6.6586 - - - - -
0.4844 90 5.5276 - - - - -
0.5382 100 5.9891 - - - - -
0.5920 110 5.2983 - - - - -
0.6458 120 5.6242 - - - - -
0.6996 130 5.498 - - - - -
0.7534 140 4.4201 - - - - -
0.8073 150 4.3818 - - - - -
0.8611 160 4.2175 - - - - -
0.9149 170 4.2341 - - - - -
0.9687 180 4.3349 - - - - -
0.9956 185 - 0.2664 0.2607 0.2508 0.2263 0.1796
1.0269 190 4.6803 - - - - -
1.0807 200 3.877 - - - - -
1.1345 210 4.0309 - - - - -
1.1884 220 4.0755 - - - - -
1.2422 230 3.9068 - - - - -
1.2960 240 4.188 - - - - -
1.3498 250 4.3417 - - - - -
1.4036 260 4.0526 - - - - -
1.4575 270 3.3933 - - - - -
1.5113 280 3.8309 - - - - -
1.5651 290 3.5633 - - - - -
1.6189 300 3.8179 - - - - -
1.6727 310 4.0671 - - - - -
1.7265 320 3.3919 - - - - -
1.7804 330 2.6578 - - - - -
1.8342 340 2.6953 - - - - -
1.8880 350 2.8858 - - - - -
1.9418 360 2.8933 - - - - -
1.9956 370 2.9603 0.2775 0.2737 0.2637 0.2402 0.1916
2.0538 380 3.3361 - - - - -
2.1076 390 2.7904 - - - - -
2.1615 400 3.0108 - - - - -
2.2153 410 2.8917 - - - - -
2.2691 420 3.0295 - - - - -
2.3229 430 3.5609 - - - - -
2.3767 440 2.7722 - - - - -
2.4305 450 3.2115 - - - - -
2.4844 460 2.6333 - - - - -
2.5382 470 3.2503 - - - - -
2.5920 480 2.7708 - - - - -
2.6458 490 3.167 - - - - -
2.6996 500 3.1447 - - - - -
2.7534 510 2.0428 - - - - -
2.8073 520 2.0001 - - - - -
2.8611 530 2.0826 - - - - -
2.9149 540 2.0853 - - - - -
2.9687 550 2.2365 - - - - -
2.9956 555 - 0.2660 0.2604 0.2509 0.2266 0.1810
3.0269 560 2.762 - - - - -
3.0807 570 2.1219 - - - - -
3.1345 580 2.2908 - - - - -
3.1884 590 2.6195 - - - - -
3.2422 600 2.3468 - - - - -
3.2960 610 2.7504 - - - - -
3.3498 620 2.9486 - - - - -
3.4036 630 2.7281 - - - - -
3.4575 640 2.188 - - - - -
3.5113 650 2.5494 - - - - -
3.5651 660 2.426 - - - - -
3.6189 670 2.6478 - - - - -
3.6727 680 2.9209 - - - - -
3.7265 690 2.3512 - - - - -
3.7804 700 1.6746 - - - - -
3.8342 710 1.739 - - - - -
3.8880 720 1.951 - - - - -
3.9418 730 1.9886 - - - - -
3.9956 740 2.1022 0.2574 0.2517 0.2405 0.2194 0.1746
  • The bold row denotes the saved checkpoint.

Environmental Impact

Carbon emissions were measured using CodeCarbon.

  • Energy Consumed: 0.205 kWh
  • Carbon Emitted: 0.011 kg of CO2
  • Hours Used: 6.806 hours

Training Hardware

  • On Cloud: No
  • GPU Model: 1 x NVIDIA GeForce GTX 1660 Ti
  • CPU Model: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  • RAM Size: 7.68 GB

Framework Versions

  • Python: 3.11.11
  • Sentence Transformers: 3.3.1
  • Transformers: 4.48.0.dev0
  • PyTorch: 2.5.1
  • Accelerate: 1.2.1
  • Datasets: 2.19.1
  • Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
0
Safetensors
Model size
149M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for Steve77/modernbert-embed-base-bible

Finetuned
(5)
this model

Evaluation results