`core.datasets.retro.index.factory`#

The IndexFactory constructs an index from an index type string.

Module Contents#

Get index.

class core.datasets.retro.index.factory.IndexFactory#

Get index.

Index type generally read from argument ‘–retro-index-ty’.

classmethod get_index_class(index_type: str) → type#

Get an index class, given a type string.

Parameters:: index_type (str) – One of ‘faiss-base’ (naive Faiss index wrapper) or ‘faiss-par-add’ (Faiss index wrapper with near embarrassingly parallel index.add().
Returns:: An Index sub-type corresponding to the index_type.

classmethod get_index( index_type: str, ) → megatron.core.datasets.retro.index.index.Index#

Construct an index from an index type string.

Parameters:: index_type (str) – One of ‘faiss-base’ (naive Faiss index wrapper) or ‘faiss-par-add’ (Faiss index wrapper with near embarrassingly parallel index.add().
Returns:: An Index instance corresponding to the index_type.