# TfidfTransformer

### *class* cuml.feature_extraction.text.TfidfTransformer(, norm='l2', use_idf=True, smooth_idf=True, sublinear_tf=False, verbose=False, output_type=None)[[source]](https://github.com/NVIDIA/cuml/blob/93418baf5/python/cuml/cuml/feature_extraction/text.py#L998)

Transform a count matrix to a normalized tf or tf-idf representation.

Tf means term-frequency while tf-idf means term-frequency times inverse
document-frequency. This is a common term weighting scheme in information
retrieval, that has also found good use in document classification.

The goal of using tf-idf instead of the raw frequencies of occurrence of a
token in a given document is to scale down the impact of tokens that occur
very frequently in a given corpus and that are hence empirically less
informative than features that occur in a small fraction of the training
corpus.

The formula that is used to compute the tf-idf for a term t of a document d
in a document set is tf-idf(t, d) = tf(t, d) \* idf(t), and the idf is
computed as idf(t) = log [ n / df(t) ] + 1 (if `smooth_idf=False`), where
n is the total number of documents in the document set and df(t) is the
document frequency of t; the document frequency is the number of documents
in the document set that contain the term t. The effect of adding “1” to
the idf in the equation above is that terms with zero idf, i.e., terms
that occur in all documents in a training set, will not be entirely
ignored.
(Note that the idf formula above differs from the standard textbook
notation that defines the idf as
idf(t) = log [ n / (df(t) + 1) ]).

If `smooth_idf=True` (the default), the constant “1” is added to the
numerator and denominator of the idf as if an extra document was seen
containing every term in the collection exactly once, which prevents
zero divisions: idf(t) = log [ (1 + n) / (1 + df(t)) ] + 1.

Furthermore, the formulas used to compute tf and idf depend
on parameter settings that correspond to the SMART notation used in IR
as follows:

Tf is “n” (natural) by default, “l” (logarithmic) when
`sublinear_tf=True`.
Idf is “t” when use_idf is given, “n” (none) otherwise.
Normalization is “c” (cosine) when `norm='l2'`, “n” (none)
when `norm=None`.

* **Parameters:**
  **norm**
  : Norm used to normalize term vectors. None for no normalization.

  **use_idf**
  : Enable inverse-document-frequency reweighting.

  **smooth_idf**
  : Smooth idf weights by adding one to document frequencies, as if an
    extra document was seen containing every term in the collection
    exactly once. Prevents zero divisions.

  **sublinear_tf**
  : Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf).

  **verbose**
  : Sets logging level. It must be one of `cuml.common.logger.level_*`.
    See [Verbosity Levels](../../cuml/index.html.md#verbosity-levels) for more info.

  **output_type**
  : Return results and set estimator attributes to the indicated output
    type. If None, the output type set at the module level
    (`cuml.global_settings.output_type`) will be used. See
    [Output Data Type Configuration](../../cuml/index.html.md#output-data-type-configuration) for more info.
* **Attributes:**
  **idf_**
  : The inverse document frequency (IDF) vector; only defined
    if  `use_idf` is True.

### Methods

| [`fit`](#cuml.feature_extraction.text.TfidfTransformer.fit)(X[, y])                 | Fit the transformer.                                     |
|------------------------------------------------------------------------------|----------------------------------------------------------|
| [`fit_transform`](#cuml.feature_extraction.text.TfidfTransformer.fit_transform)(X[, y, copy]) | Fit the transformer, then transform X.                   |
| [`transform`](#cuml.feature_extraction.text.TfidfTransformer.transform)(X[, copy])        | Transform a count matrix to tf or tf-idf representation. |

### Examples

```pycon
>>> from cuml.feature_extraction.text import TfidfTransformer
>>> from cuml.feature_extraction.text import CountVectorizer
>>> from sklearn.pipeline import Pipeline
>>> corpus = ['this is the first document',
...           'this document is the second document',
...           'and this is the third one',
...           'is this the first document']
>>> pipe = Pipeline([('count', CountVectorizer()),
...                  ('tfid', TfidfTransformer())])
>>> X = pipe.fit_transform(corpus)
>>> X.shape
(4, 9)
```

<!-- !! processed by numpydoc !! -->

#### fit(X, y=None)[[source]](https://github.com/NVIDIA/cuml/blob/93418baf5/python/cuml/cuml/feature_extraction/text.py#L1188)

Fit the transformer.

* **Parameters:**
  **X**
  : A matrix of term/token counts.

  **y**
  : Ignored. Exists for API compatibility only.
* **Returns:**
  **self**
  : The instance itself.

<!-- !! processed by numpydoc !! -->

#### fit_transform(X, y=None, copy=True)[[source]](https://github.com/NVIDIA/cuml/blob/93418baf5/python/cuml/cuml/feature_extraction/text.py#L1209)

Fit the transformer, then transform X.

* **Parameters:**
  **X**
  : A matrix of term/token counts.

  **y**
  : Ignored. Exists for API compatibility only.

  **copy**
  : If `copy=False,` then [`fit_transform`](#cuml.feature_extraction.text.TfidfTransformer.fit_transform) may choose to mutate `X`
    in-place if that would be more efficient.
* **Returns:**
  **X**
  : Tf-idf weighted document-term matrix.

<!-- !! processed by numpydoc !! -->

#### get_feature_names_out(input_features=None)

Get output feature names for transformation.

* **Parameters:**
  **input_features**
  : Input features.
    - If `input_features` is [`None`](https://docs.python.org/3/builtins/constants.html#None), then `feature_names_in_` is
      used as feature names in. If `feature_names_in_` is not defined,
      then the following input feature names are generated:
      `["x0", "x1", ..., "x(n_features_in_ - 1)"]`.
    - If `input_features` is an array-like, then `input_features` must
      match `feature_names_in_` if `feature_names_in_` is defined.
* **Returns:**
  **feature_names_out**
  : Same as input features.

<!-- !! processed by numpydoc !! -->

#### get_params(deep=True)[[source]](https://github.com/NVIDIA/cuml/blob/93418baf5/python/cuml/cuml/internals/base.py#L191)

Returns a dict of all params owned by this class. If the child class
has appropriately overridden the `_get_param_names` method and does not
need anything other than what is there in this method, then it doesn’t
have to override this method

<!-- !! processed by numpydoc !! -->

#### set_params(\*\*params)[[source]](https://github.com/NVIDIA/cuml/blob/93418baf5/python/cuml/cuml/internals/base.py#L200)

Accepts a dict of params and updates the corresponding ones owned by
this class. If the child class has appropriately overridden the
`_get_param_names` method and does not need anything other than what is,
there in this method, then it doesn’t have to override this method

<!-- !! processed by numpydoc !! -->

#### transform(X, copy=True)[[source]](https://github.com/NVIDIA/cuml/blob/93418baf5/python/cuml/cuml/feature_extraction/text.py#L1234)

Transform a count matrix to tf or tf-idf representation.

* **Parameters:**
  **X**
  : A matrix of term/token counts.

  **copy**
  : If `copy=False,` then [`fit_transform`](#cuml.feature_extraction.text.TfidfTransformer.fit_transform) may choose to mutate `X`
    in-place if that would be more efficient.
* **Returns:**
  **X**
  : Tf-idf weighted document-term matrix.

<!-- !! processed by numpydoc !! -->