Is this page helpful?

CountVectorizer#

class cuml.feature_extraction.text.CountVectorizer( *, lowercase=True, preprocessor=None, tokenizer=None, delimiter=None, stop_words=None, ngram_range=(1, 1), analyzer='word', max_df=1.0, min_df=1, max_features=None, vocabulary=None, binary=False, dtype=<class 'numpy.float32'>, verbose=False, output_type=None, )[source]#

Convert a collection of text documents to a matrix of token counts.

If you do not provide an a-priori dictionary then the number of features will be equal to the vocabulary size found by analyzing the data.

Parameters:

lowercasebool, default=True

Convert all characters to lowercase before tokenizing.

preprocessorcallable, default=None

Override the preprocessing (string transformation) stage while preserving the tokenizing and n-grams generation steps. This function receives a cudf.Series of strings and should return a cudf.Series of strings.

tokenizercallable, default=None

Override the string tokenization step while preserving the preprocessing and n-grams generation steps. This function receives a cudf.Series of strings and should return a cudf.Series of lists of strings. Only applies if analyzer == 'word'.

delimiterstr, default=None

String used to delimit tokens in the document. If None, then any non-alphanumeric (or “ “) character is treated as a delimiter. Only applies if analyzer == "word'.

stop_words{‘english’}, list, default=None

If ‘english’, a built-in stop word list for English is used. If a list, that list is assumed to contain stop words, all of which will be removed from the resulting tokens. If None, no stop words will be used. Only applies if analyzer == 'word'.

ngram_rangetuple (min_n, max_n), default=(1, 1)

The lower and upper boundary of the range of n-values for different n-grams to be extracted. All values of n such that min_n <= n <= max_n will be used. For example an ngram_range of (1, 1) means only unigrams, (1, 2) means unigrams and bigrams, and (2, 2) means only bigrams.

analyzer{‘word’, ‘char’, ‘char_wb’}, default=’word’

Whether the feature should be made of word or character n-grams. Option ‘char_wb’ creates character n-grams only from text inside word boundaries; n-grams at the edges of words are padded with space.

max_dffloat in range [0.0, 1.0] or int, default=1.0

When building the vocabulary ignore terms that have a document frequency strictly higher than the given threshold (corpus-specific stop words). If float, the parameter represents a proportion of documents, integer absolute counts. This parameter is ignored if vocabulary is not None.

min_dffloat in range [0.0, 1.0] or int, default=1

When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold. This value is also called cut-off in the literature. If float, the parameter represents a proportion of documents, integer absolute counts. This parameter is ignored if vocabulary is not None.

max_featuresint, default=None

If not None, build a vocabulary that only consider the top max_features ordered by term frequency across the corpus. Otherwise, all features are used.

This parameter is ignored if vocabulary is not None.

vocabularyarray-like or mapping, default=None

Either an array-like of terms, or a mapping where keys are terms and values are indices in the feature matrix. If not given, a vocabulary is determined from the input documents.

binarybool, default=False

If True, all non zero counts are set to 1. This is useful for discrete probabilistic models that model binary events rather than integer counts.

dtypedtype, default=np.float32

Type of the matrix returned by fit_transform() or transform().

verboseint or boolean, default=False

Sets logging level. It must be one of cuml.common.logger.level_*. See Verbosity Levels for more info.

output_type{None, ‘input’, ‘cupy’, ‘numpy’, ‘cudf’, ‘pandas’}, default=None

Return results and set estimator attributes to the indicated output type. If None, the output type set at the module level (cuml.global_settings.output_type) will be used. See Output Data Type Configuration for more info.

Attributes:

vocabulary_cudf.Series: The vocabulary used to map terms to feature indices.
fixed_vocabulary_bool: True if a fixed vocabulary of term to indices mapping is provided by the user.

Methods

`fit`(X[, y])	Fit the vectorizer.
`fit_transform`(X[, y])	Fit the vectorizer and return a document-term matrix.
`get_feature_names_out`([input_features])	Get output feature names for transformation.
`inverse_transform`(X)	Return terms per document with nonzero entries in X.
`transform`(X)	Transform documents to document-term matrix.

Examples

>>> from cuml.feature_extraction.text import CountVectorizer
>>> corpus = [
...     'This is the first document.',
...     'This document is the second document.',
...     'And this is the third one.',
...     'Is this the first document?',
... ]
>>> vectorizer = CountVectorizer()
>>> X = vectorizer.fit_transform(corpus)
>>> vectorizer.get_feature_names_out()
array(['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third',
       'this'], ...)
>>> X.shape
(4, 9)

fit(X, y=None)[source]#

Fit the vectorizer.

Parameters:

XIterable[str]: Training samples. Each sample must be a text document which will be tokenized and hashed.
yNone: Ignored. Exists for API compatibility only.

Returns:

selfobject: The instance itself.

fit_transform(X, y=None)[source]#

Fit the vectorizer and return a document-term matrix.

Parameters:

XIterable[str]: Training samples. Each sample must be a text document which will be tokenized and hashed.
yNone: Ignored. Exists for API compatibility only.

Returns:

Xsparse matrix of shape (n_samples, n_features): Document-term matrix.

get_feature_names(input_features=None)[source]#: Return feature names for output features.

Deprecated since version 26.10: This method was deprecated in version 26.10 and will be removed in version 26.12. Please use get_feature_names_out instead.

get_feature_names_out(input_features=None)[source]#

Get output feature names for transformation.

Parameters:

input_featuresarray-like of str or None, default=None: Not used, present here for API consistency by convention.

Returns:

feature_names_outnumpy.ndarray of str objects.: Transformed feature names.

get_params(deep=True)[source]#: Returns a dict of all params owned by this class. If the child class has appropriately overridden the _get_param_names method and does not need anything other than what is there in this method, then it doesn’t have to override this method

inverse_transform(X)[source]#

Return terms per document with nonzero entries in X.

Parameters:

Xarray-like or sparse matrix, shape=(n_samples, n_features): Document-term matrix.

Returns:

X_originallist of arrays of shape (n_samples,): List of arrays of terms.

set_params(**params)[source]#: Accepts a dict of params and updates the corresponding ones owned by this class. If the child class has appropriately overridden the _get_param_names method and does not need anything other than what is, there in this method, then it doesn’t have to override this method

transform(X)[source]#

Transform documents to document-term matrix.

Parameters:

XIterable[str]: Training samples. Each sample must be a text document which will be tokenized and hashed.

Returns:

Xsparse matrix of shape (n_samples, n_features): Document-term matrix.