SBERT#

Sentence-BERT (SBERT) is a variant of BERT (Bidirectional Encoder Representations from Transformers) designed to generate fixed-size sentence embeddings. Unlike the original BERT model, which produces token-level embeddings, SBERT fine-tunes BERT to produce meaningful, dense representations of entire sentences. These embeddings can be used for tasks like sentence similarity, clustering, and information retrieval, where comparing sentence-level meaning is key. SBERT improves efficiency by enabling fast, scalable comparisons of sentences using cosine similarity or other distance metrics.

NeMo 2.0 Finetuning Recipes#

Note

The finetuning recipes use the SpecterDataModule for the data argument. You can replace the SpecterDataModule with your custom dataset.

To import the HF model and convert to NeMo 2.0 format, run the following command (this only needs to be done once)

from nemo.collections import llm
llm.import_ckpt(model=llm.BertEmbeddingModel(llm.BertEmbeddingLargeConfig()), source='hf://intfloat/e5-large-v2')

We provide an example below on how to invoke the default recipe and override the data argument:

from nemo.collections import llm

recipe = llm.recipes.e5_340m.finetune_recipe(
    name="e5_large_finetuning",
    resume_path=f"/path/to/checkpoints",
    num_nodes=1,
    num_gpus_per_node=8,
)

# # To override the data argument
# dataloader = a_function_that_configures_your_custom_dataset(
#     gbs=gbs,
#     mbs=mbs,
#     seq_length=recipe.model.config.seq_length,
# )
# recipe.data = dataloader

Currently SBERT model only supports SFT.

Note

The configuration in the recipes is done using the NeMo-Run run.Config and run.Partial configuration objects. Please review the NeMo-Run documentation to learn more about its configuration and execution system.

Once you have your final configuration ready, you can execute it on any of the NeMo-Run supported executors. The simplest is the local executor, which just runs the pretraining locally in a separate process. You can use it as follows:

import nemo_run as run

run.run(recipe, executor=run.LocalExecutor())

Additionally, you can also run it directly in the same Python process as follows:

run.run(recipe, direct=True)

Recipe

Status

E5-Large (340M)

Yes
On this page