morpheus.models.dfencoder.dataframe.EncoderDataFrame

Bases: pandas.core.frame.DataFrame

Attributes

T

at

Access a single value for a row/column label pair.

attrs

Dictionary of global attributes of this dataset.

axes

Return a list representing the axes of the DataFrame.

columns

The column labels of the DataFrame.

dtypes

Return the dtypes in the DataFrame.

empty

Indicator whether DataFrame is empty.

flags

Get the properties associated with this pandas object.

iat

Access a single value for a row/column pair by integer position.

iloc

Purely integer-location based indexing for selection by position.

index

The index (row labels) of the DataFrame.

loc

Access a group of rows and columns by label(s) or a boolean array.

ndim

Return an int representing the number of axes / array dimensions.

shape

Return a tuple representing the dimensionality of the DataFrame.

size

Return an int representing the number of elements in this object.

style

Returns a Styler object.

values

Return a Numpy representation of the DataFrame.

Methods

abs()[source]

Return a Series/DataFrame with absolute numeric value of each element.

This function only applies to elements that are all numeric.

Returns

abs

Series/DataFrame containing the absolute value of each element.

See also

numpy.absolute

Calculate the absolute value element-wise.

Notes

For complex inputs, 1.2 + 1j, the absolute value is \(\sqrt{ a^2 + b^2 }\).

Examples

Absolute numeric values in a Series.

Copy

Copied!

            

            
>>> s = pd.Series([-1.10, 2, -3.33, 4])
>>> s.abs()
0    1.10
1    2.00
2    3.33
3    4.00
dtype: float64
        

Absolute numeric values in a Series with complex numbers.

Copy

Copied!

            

            
>>> s = pd.Series([1.2 + 1j])
>>> s.abs()
0    1.56205
dtype: float64
        

Absolute numeric values in a Series with a Timedelta element.

Copy

Copied!

            

            
>>> s = pd.Series([pd.Timedelta('1 days')])
>>> s.abs()
0   1 days
dtype: timedelta64[ns]
        

Select rows with data closest to certain value using argsort (from StackOverflow).

Copy

Copied!

            

            
>>> df = pd.DataFrame({
...     'a': [4, 5, 6, 7],
...     'b': [10, 20, 30, 40],
...     'c': [100, 50, -30, -50]
... })
>>> df
a    b    c
0    4   10  100
1    5   20   50
2    6   30  -30
3    7   40  -50
>>> df.loc[(df.c - 43).abs().argsort()]
a    b    c
1    5   20   50
0    4   10  100
2    6   30  -30
3    7   40  -50
        

add(other, axis='columns', level=None, fill_value=None)[source]

Get Addition of dataframe and other, element-wise (binary operator add).

Equivalent to dataframe + other, but with support to substitute a fill_value for missing data in one of the inputs. With reverse version, radd.

Among flexible wrappers (add, sub, mul, div, mod, pow) to arithmetic operators: +, -, *, /, //, %, **.

Parameters

otherscalar, sequence, Series, or DataFrame

Any single or multiple element data structure, or list-like object.

axis{0 or ‘index’, 1 or ‘columns’}

Whether to compare by the index (0 or ‘index’) or columns (1 or ‘columns’). For Series input, axis to match Series index on.

levelint or label

Broadcast across a level, matching Index values on the passed MultiIndex level.

fill_valuefloat or None, default None

Fill existing missing (NaN) values, and any new element needed for successful DataFrame alignment, with this value before computation. If data in both corresponding DataFrame locations is missing the result will be missing.

Returns

DataFrame

Result of the arithmetic operation.

See also

DataFrame.add

Add DataFrames.

DataFrame.sub

Subtract DataFrames.

DataFrame.mul

Multiply DataFrames.

DataFrame.div

Divide DataFrames (float division).

DataFrame.truediv

Divide DataFrames (float division).

DataFrame.floordiv

Divide DataFrames (integer division).

DataFrame.mod

Calculate modulo (remainder after division).

DataFrame.pow

Calculate exponential power.

Notes

Mismatched indices will be unioned together.

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame({'angles': [0, 3, 4],
...                    'degrees': [360, 180, 360]},
...                   index=['circle', 'triangle', 'rectangle'])
>>> df
angles  degrees
circle          0      360
triangle        3      180
rectangle       4      360
        

Add a scalar with operator version which return the same results.

Copy

Copied!

            

            
>>> df + 1
angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361
        

Copy

Copied!

            

            
>>> df.add(1)
angles  degrees
circle          1      361
triangle        4      181
rectangle       5      361
        

Divide by constant with reverse version.

Copy

Copied!

            

            
>>> df.div(10)
angles  degrees
circle        0.0     36.0
triangle      0.3     18.0
rectangle     0.4     36.0
        

Copy

Copied!

            

            
>>> df.rdiv(10)
angles   degrees
circle          inf  0.027778
triangle   3.333333  0.055556
rectangle  2.500000  0.027778
        

Subtract a list and Series by axis with operator version.

Copy

Copied!

            

            
>>> df - [1, 2]
angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358
        

Copy

Copied!

            

            
>>> df.sub([1, 2], axis='columns')
angles  degrees
circle         -1      358
triangle        2      178
rectangle       3      358
        

Copy

Copied!

            

            
>>> df.sub(pd.Series([1, 1, 1], index=['circle', 'triangle', 'rectangle']),
...        axis='index')
angles  degrees
circle         -1      359
triangle        2      179
rectangle       3      359
        

Multiply a DataFrame of different shape with operator version.

Copy

Copied!

            

            
>>> other = pd.DataFrame({'angles': [0, 3, 4]},
...                      index=['circle', 'triangle', 'rectangle'])
>>> other
angles
circle          0
triangle        3
rectangle       4
        

Copy

Copied!

            

            
>>> df * other
angles  degrees
circle          0      NaN
triangle        9      NaN
rectangle      16      NaN
        

Copy

Copied!

            

            
>>> df.mul(other, fill_value=0)
angles  degrees
circle          0      0.0
triangle        9      0.0
rectangle      16      0.0
        

Divide by a MultiIndex by level.

Copy

Copied!

            

            
>>> df_multindex = pd.DataFrame({'angles': [0, 3, 4, 4, 5, 6],
...                              'degrees': [360, 180, 360, 360, 540, 720]},
...                             index=[['A', 'A', 'A', 'B', 'B', 'B'],
...                                    ['circle', 'triangle', 'rectangle',
...                                     'square', 'pentagon', 'hexagon']])
>>> df_multindex
angles  degrees
A circle          0      360
triangle        3      180
rectangle       4      360
B square          4      360
pentagon        5      540
hexagon         6      720
        

Copy

Copied!

            

            
>>> df.div(df_multindex, level=1, fill_value=0)
angles  degrees
A circle        NaN      1.0
triangle      1.0      1.0
rectangle     1.0      1.0
B square        0.0      0.0
pentagon      0.0      0.0
hexagon       0.0      0.0
        

add_prefix(prefix)[source]

Prefix labels with string prefix.

For Series, the row labels are prefixed. For DataFrame, the column labels are prefixed.

Parameters

prefixstr

The string to add before each label.

Returns

Series or DataFrame

New Series or DataFrame with updated labels.

See also

Series.add_suffix

Suffix row labels with string suffix.

DataFrame.add_suffix

Suffix column labels with string suffix.

Examples

Copy

Copied!

            

            
>>> s = pd.Series([1, 2, 3, 4])
>>> s
0    1
1    2
2    3
3    4
dtype: int64
        

Copy

Copied!

            

            
>>> s.add_prefix('item_')
item_0    1
item_1    2
item_2    3
item_3    4
dtype: int64
        

Copy

Copied!

            

            
>>> df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
>>> df
A  B
0  1  3
1  2  4
2  3  5
3  4  6
        

Copy

Copied!

            

            
>>> df.add_prefix('col_')
col_A  col_B
0       1       3
1       2       4
2       3       5
3       4       6
        

add_suffix(suffix)[source]

Suffix labels with string suffix.

For Series, the row labels are suffixed. For DataFrame, the column labels are suffixed.

Parameters

suffixstr

The string to add after each label.

Returns

Series or DataFrame

New Series or DataFrame with updated labels.

See also

Series.add_prefix

Prefix row labels with string prefix.

DataFrame.add_prefix

Prefix column labels with string prefix.

Examples

Copy

Copied!

            

            
>>> s = pd.Series([1, 2, 3, 4])
>>> s
0    1
1    2
2    3
3    4
dtype: int64
        

Copy

Copied!

            

            
>>> s.add_suffix('_item')
0_item    1
1_item    2
2_item    3
3_item    4
dtype: int64
        

Copy

Copied!

            

            
>>> df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]})
>>> df
A  B
0  1  3
1  2  4
2  3  5
3  4  6
        

Copy

Copied!

            

            
>>> df.add_suffix('_col')
A_col  B_col
0       1       3
1       2       4
2       3       5
3       4       6
        

agg(func=None, axis=0, *args, **kwargs)[source]

Aggregate using one or more operations over the specified axis.

Parameters

funcfunction, str, list or dict

Function to use for aggregating the data. If a function, must either work when passed a DataFrame or when passed to DataFrame.apply.

Accepted combinations are:

• function

• string function name

• list of functions and/or function names, e.g. [np.sum, 'mean']

• dict of axis labels -> functions, function names or list of such.

axis{0 or ‘index’, 1 or ‘columns’}, default 0

If 0 or ‘index’: apply function to each column. If 1 or ‘columns’: apply function to each row.

*args

Positional arguments to pass to func.

**kwargs

Keyword arguments to pass to func.

Returns

scalar, Series or DataFrame

The return can be:

• scalar : when Series.agg is called with single function

• Series : when DataFrame.agg is called with a single function

• DataFrame : when DataFrame.agg is called with several functions

Return scalar, Series or DataFrame.

The aggregation operations are always performed over an axis, either the

index (default) or the column axis. This behavior is different from

numpy aggregation functions (mean, median, prod, sum, std,

var), where the default is to compute the aggregation of the flattened

array, e.g., numpy.mean(arr_2d) as opposed to

numpy.mean(arr_2d, axis=0).

agg is an alias for aggregate. Use the alias.

See also

DataFrame.apply

Perform any type of operations.

DataFrame.transform

Perform transformation type operations.

core.groupby.GroupBy

Perform operations over groups.

core.resample.Resampler

Perform operations over resampled bins.

core.window.Rolling

Perform operations over rolling window.

core.window.Expanding

Perform operations over expanding window.

core.window.ExponentialMovingWindow

Perform operation over exponential weighted window.

Notes

agg is an alias for aggregate. Use the alias.

Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See gotchas.udf-mutation for more details.

A passed user-defined-function will be passed a Series for evaluation.

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame([[1, 2, 3],
...                    [4, 5, 6],
...                    [7, 8, 9],
...                    [np.nan, np.nan, np.nan]],
...                   columns=['A', 'B', 'C'])
        

Aggregate these functions over the rows.

Copy

Copied!

            

            
>>> df.agg(['sum', 'min'])
A     B     C
sum  12.0  15.0  18.0
min   1.0   2.0   3.0
        

Different aggregations per column.

Copy

Copied!

            

            
>>> df.agg({'A' : ['sum', 'min'], 'B' : ['min', 'max']})
A    B
sum  12.0  NaN
min   1.0  2.0
max   NaN  8.0
        

Aggregate different functions over the columns and rename the index of the resulting DataFrame.

Copy

Copied!

            

            
>>> df.agg(x=('A', max), y=('B', 'min'), z=('C', np.mean))
A    B    C
x  7.0  NaN  NaN
y  NaN  2.0  NaN
z  NaN  NaN  6.0
        

Aggregate over the columns.

Copy

Copied!

            

            
>>> df.agg("mean", axis="columns")
0    2.0
1    5.0
2    8.0
3    NaN
dtype: float64
        

aggregate(func=None, axis=0, *args, **kwargs)[source]

Aggregate using one or more operations over the specified axis.

Parameters

funcfunction, str, list or dict

Function to use for aggregating the data. If a function, must either work when passed a DataFrame or when passed to DataFrame.apply.

Accepted combinations are:

• function

• string function name

• list of functions and/or function names, e.g. [np.sum, 'mean']

• dict of axis labels -> functions, function names or list of such.

axis{0 or ‘index’, 1 or ‘columns’}, default 0

If 0 or ‘index’: apply function to each column. If 1 or ‘columns’: apply function to each row.

*args

Positional arguments to pass to func.

**kwargs

Keyword arguments to pass to func.

Returns

scalar, Series or DataFrame

The return can be:

• scalar : when Series.agg is called with single function

• Series : when DataFrame.agg is called with a single function

• DataFrame : when DataFrame.agg is called with several functions

Return scalar, Series or DataFrame.

The aggregation operations are always performed over an axis, either the

index (default) or the column axis. This behavior is different from

numpy aggregation functions (mean, median, prod, sum, std,

var), where the default is to compute the aggregation of the flattened

array, e.g., numpy.mean(arr_2d) as opposed to

numpy.mean(arr_2d, axis=0).

agg is an alias for aggregate. Use the alias.

See also

DataFrame.apply

Perform any type of operations.

DataFrame.transform

Perform transformation type operations.

core.groupby.GroupBy

Perform operations over groups.

core.resample.Resampler

Perform operations over resampled bins.

core.window.Rolling

Perform operations over rolling window.

core.window.Expanding

Perform operations over expanding window.

core.window.ExponentialMovingWindow

Perform operation over exponential weighted window.

Notes

agg is an alias for aggregate. Use the alias.

Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See gotchas.udf-mutation for more details.

A passed user-defined-function will be passed a Series for evaluation.

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame([[1, 2, 3],
...                    [4, 5, 6],
...                    [7, 8, 9],
...                    [np.nan, np.nan, np.nan]],
...                   columns=['A', 'B', 'C'])
        

Aggregate these functions over the rows.

Copy

Copied!

            

            
>>> df.agg(['sum', 'min'])
A     B     C
sum  12.0  15.0  18.0
min   1.0   2.0   3.0
        

Different aggregations per column.

Copy

Copied!

            

            
>>> df.agg({'A' : ['sum', 'min'], 'B' : ['min', 'max']})
A    B
sum  12.0  NaN
min   1.0  2.0
max   NaN  8.0
        

Aggregate different functions over the columns and rename the index of the resulting DataFrame.

Copy

Copied!

            

            
>>> df.agg(x=('A', max), y=('B', 'min'), z=('C', np.mean))
A    B    C
x  7.0  NaN  NaN
y  NaN  2.0  NaN
z  NaN  NaN  6.0
        

Aggregate over the columns.

Copy

Copied!

            

            
>>> df.agg("mean", axis="columns")
0    2.0
1    5.0
2    8.0
3    NaN
dtype: float64
        

align(other, join='outer', axis=None, level=None, copy=True, fill_value=None, method=None, limit=None, fill_axis=0, broadcast_axis=None)[source]

Align two objects on their axes with the specified join method.

Join method is specified for each axis Index.

Parameters

otherDataFrame or Series

join{‘outer’, ‘inner’, ‘left’, ‘right’}, default ‘outer’

axisallowed axis of the other object, default None

Align on index (0), columns (1), or both (None).

levelint or level name, default None

Broadcast across a level, matching Index values on the passed MultiIndex level.

copybool, default True

Always returns new objects. If copy=False and no reindexing is required then original objects are returned.

fill_valuescalar, default np.NaN

Value to use for missing values. Defaults to NaN, but can be any “compatible” value.

method{‘backfill’, ‘bfill’, ‘pad’, ‘ffill’, None}, default None

Method to use for filling holes in reindexed Series:

• pad / ffill: propagate last valid observation forward to next valid.

• backfill / bfill: use NEXT valid observation to fill gap.

limitint, default None

If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill. In other words, if there is a gap with more than this number of consecutive NaNs, it will only be partially filled. If method is not specified, this is the maximum number of entries along the entire axis where NaNs will be filled. Must be greater than 0 if not None.

fill_axis{0 or ‘index’, 1 or ‘columns’}, default 0

Filling axis, method and limit.

broadcast_axis{0 or ‘index’, 1 or ‘columns’}, default None

Broadcast values along this axis, if aligning two objects of different dimensions.

Returns

(left, right)(DataFrame, type of other)

Aligned objects.

all(axis=0, bool_only=None, skipna=True, level=None, **kwargs)[source]

Return whether all elements are True, potentially over an axis.

Returns True unless there at least one element within a series or along a Dataframe axis that is False or equivalent (e.g. zero or empty).

Parameters

axis{0 or ‘index’, 1 or ‘columns’, None}, default 0

Indicate which axis or axes should be reduced.

• 0 / ‘index’ : reduce the index, return a Series whose index is the original column labels.

• 1 / ‘columns’ : reduce the columns, return a Series whose index is the original index.

• None : reduce all axes, return a scalar.

bool_onlybool, default None

Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for Series.

skipnabool, default True

Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be True, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero.

levelint or level name, default None

If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.

**kwargsany, default None

Additional keywords have no effect but might be accepted for compatibility with NumPy.

Returns

Series or DataFrame

If level is specified, then, DataFrame is returned; otherwise, Series is returned.

See also

Series.all

Return True if all elements are True.

DataFrame.any

Return True if one (or more) elements are True.

Examples

Series

Copy

Copied!

            

            
>>> pd.Series([True, True]).all()
True
>>> pd.Series([True, False]).all()
False
>>> pd.Series([], dtype="float64").all()
True
>>> pd.Series([np.nan]).all()
True
>>> pd.Series([np.nan]).all(skipna=False)
True
        

DataFrames

Create a dataframe from a dictionary.

Copy

Copied!

            

            
>>> df = pd.DataFrame({'col1': [True, True], 'col2': [True, False]})
>>> df
col1   col2
0  True   True
1  True  False
        

Default behaviour checks if column-wise values all return True.

Copy

Copied!

            

            
>>> df.all()
col1     True
col2    False
dtype: bool
        

Specify axis='columns' to check if row-wise values all return True.

Copy

Copied!

            

            
>>> df.all(axis='columns')
0     True
1    False
dtype: bool
        

Or axis=None for whether every value is True.

Copy

Copied!

            

            
>>> df.all(axis=None)
False
        

any(axis=0, bool_only=None, skipna=True, level=None, **kwargs)[source]

Return whether any element is True, potentially over an axis.

Returns False unless there is at least one element within a series or along a Dataframe axis that is True or equivalent (e.g. non-zero or non-empty).

Parameters

axis{0 or ‘index’, 1 or ‘columns’, None}, default 0

Indicate which axis or axes should be reduced.

• 0 / ‘index’ : reduce the index, return a Series whose index is the original column labels.

• 1 / ‘columns’ : reduce the columns, return a Series whose index is the original index.

• None : reduce all axes, return a scalar.

bool_onlybool, default None

Include only boolean columns. If None, will attempt to use everything, then use only boolean data. Not implemented for Series.

skipnabool, default True

Exclude NA/null values. If the entire row/column is NA and skipna is True, then the result will be False, as for an empty row/column. If skipna is False, then NA are treated as True, because these are not equal to zero.

levelint or level name, default None

If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a Series.

**kwargsany, default None

Additional keywords have no effect but might be accepted for compatibility with NumPy.

Returns

Series or DataFrame

If level is specified, then, DataFrame is returned; otherwise, Series is returned.

See also

numpy.any

Numpy version of this method.

Series.any

Return whether any element is True.

Series.all

Return whether all elements are True.

DataFrame.any

Return whether any element is True over requested axis.

DataFrame.all

Return whether all elements are True over requested axis.

Examples

Series

For Series input, the output is a scalar indicating whether any element is True.

Copy

Copied!

            

            
>>> pd.Series([False, False]).any()
False
>>> pd.Series([True, False]).any()
True
>>> pd.Series([], dtype="float64").any()
False
>>> pd.Series([np.nan]).any()
False
>>> pd.Series([np.nan]).any(skipna=False)
True
        

DataFrame

Whether each column contains at least one True element (the default).

Copy

Copied!

            

            
>>> df = pd.DataFrame({"A": [1, 2], "B": [0, 2], "C": [0, 0]})
>>> df
A  B  C
0  1  0  0
1  2  2  0
        

Copy

Copied!

            

            
>>> df.any()
A     True
B     True
C    False
dtype: bool
        

Aggregating over the columns.

Copy

Copied!

            

            
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 2]})
>>> df
A  B
0   True  1
1  False  2
        

Copy

Copied!

            

            
>>> df.any(axis='columns')
0    True
1    True
dtype: bool
        

Copy

Copied!

            

            
>>> df = pd.DataFrame({"A": [True, False], "B": [1, 0]})
>>> df
A  B
0   True  1
1  False  0
        

Copy

Copied!

            

            
>>> df.any(axis='columns')
0    True
1    False
dtype: bool
        

Aggregating over the entire DataFrame with axis=None.

Copy

Copied!

            

            
>>> df.any(axis=None)
True
        

any for an empty DataFrame is an empty Series.

Copy

Copied!

            

            
>>> pd.DataFrame([]).any()
Series([], dtype: bool)
        

append(other, ignore_index=False, verify_integrity=False, sort=False)[source]

Append rows of other to the end of caller, returning a new object.

Columns in other that are not in the caller are added as new columns.

Parameters

otherDataFrame or Series/dict-like object, or list of these

The data to append.

ignore_indexbool, default False

If True, the resulting axis will be labeled 0, 1, …, n - 1.

verify_integritybool, default False

If True, raise ValueError on creating index with duplicates.

sortbool, default False

Sort columns if the columns of self and other are not aligned.

Changed in version 1.0.0:Changed to not sort by default.

Returns

DataFrame

A new DataFrame consisting of the rows of caller and the rows of other.

See also

concat

General function to concatenate DataFrame or Series objects.

Notes

If a list of dict/series is passed and the keys are all contained in the DataFrame’s index, the order of the columns in the resulting DataFrame will be unchanged.

Iteratively appending rows to a DataFrame can be more computationally intensive than a single concatenate. A better solution is to append those rows to a list and then concatenate the list with the original DataFrame all at once.

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'), index=['x', 'y'])
>>> df
A  B
x  1  2
y  3  4
>>> df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'), index=['x', 'y'])
>>> df.append(df2)
A  B
x  1  2
y  3  4
x  5  6
y  7  8
        

With ignore_index set to True:

Copy

Copied!

            

            
>>> df.append(df2, ignore_index=True)
A  B
0  1  2
1  3  4
2  5  6
3  7  8
        

The following, while not recommended methods for generating DataFrames, show two ways to generate a DataFrame from multiple data sources.

Less efficient:

Copy

Copied!

            

            
>>> df = pd.DataFrame(columns=['A'])
>>> for i in range(5):
...     df = df.append({'A': i}, ignore_index=True)
>>> df
A
0  0
1  1
2  2
3  3
4  4
        

More efficient:

Copy

Copied!

            

            
>>> pd.concat([pd.DataFrame([i], columns=['A']) for i in range(5)],
...           ignore_index=True)
A
0  0
1  1
2  2
3  3
4  4
        

apply(func, axis=0, raw=False, result_type=None, args=(), **kwargs)[source]

Apply a function along an axis of the DataFrame.

Objects passed to the function are Series objects whose index is either the DataFrame’s index (axis=0) or the DataFrame’s columns (axis=1). By default (result_type=None), the final return type is inferred from the return type of the applied function. Otherwise, it depends on the result_type argument.

Parameters

funcfunction

Function to apply to each column or row.

axis{0 or ‘index’, 1 or ‘columns’}, default 0

Axis along which the function is applied:

• 0 or ‘index’: apply function to each column.

• 1 or ‘columns’: apply function to each row.

rawbool, default False

Determines if row or column is passed as a Series or ndarray object:

• False : passes each row or column as a Series to the function.

• True : the passed function will receive ndarray objects instead. If you are just applying a NumPy reduction function this will achieve much better performance.

result_type{‘expand’, ‘reduce’, ‘broadcast’, None}, default None

These only act when axis=1 (columns):

• ‘expand’ : list-like results will be turned into columns.

• ‘reduce’ : returns a Series if possible rather than expanding list-like results. This is the opposite of ‘expand’.

• ‘broadcast’ : results will be broadcast to the original shape of the DataFrame, the original index and columns will be retained.

The default behaviour (None) depends on the return value of the applied function: list-like results will be returned as a Series of those. However if the apply function returns a Series these are expanded to columns.

argstuple

Positional arguments to pass to func in addition to the array/series.

**kwargs

Additional keyword arguments to pass as keywords arguments to func.

Returns

Series or DataFrame

Result of applying func along the given axis of the DataFrame.

See also

DataFrame.applymap

For elementwise operations.

DataFrame.aggregate

Only perform aggregating type operations.

DataFrame.transform

Only perform transforming type operations.

Notes

Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See gotchas.udf-mutation for more details.

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame([[4, 9]] * 3, columns=['A', 'B'])
>>> df
A  B
0  4  9
1  4  9
2  4  9
        

Using a numpy universal function (in this case the same as np.sqrt(df)):

Copy

Copied!

            

            
>>> df.apply(np.sqrt)
A    B
0  2.0  3.0
1  2.0  3.0
2  2.0  3.0
        

Using a reducing function on either axis

Copy

Copied!

            

            
>>> df.apply(np.sum, axis=0)
A    12
B    27
dtype: int64
        

Copy

Copied!

            

            
>>> df.apply(np.sum, axis=1)
0    13
1    13
2    13
dtype: int64
        

Returning a list-like will result in a Series

Copy

Copied!

            

            
>>> df.apply(lambda x: [1, 2], axis=1)
0    [1, 2]
1    [1, 2]
2    [1, 2]
dtype: object
        

Passing result_type='expand' will expand list-like results to columns of a Dataframe

Copy

Copied!

            

            
>>> df.apply(lambda x: [1, 2], axis=1, result_type='expand')
0  1
0  1  2
1  1  2
2  1  2
        

Returning a Series inside the function is similar to passing result_type='expand'. The resulting column names will be the Series index.

Copy

Copied!

            

            
>>> df.apply(lambda x: pd.Series([1, 2], index=['foo', 'bar']), axis=1)
foo  bar
0    1    2
1    1    2
2    1    2
        

Passing result_type='broadcast' will ensure the same shape result, whether list-like or scalar is returned by the function, and broadcast it along the axis. The resulting column names will be the originals.

Copy

Copied!

            

            
>>> df.apply(lambda x: [1, 2], axis=1, result_type='broadcast')
A  B
0  1  2
1  1  2
2  1  2
        

applymap(func, na_action=None, **kwargs)[source]

Apply a function to a Dataframe elementwise.

This method applies a function that accepts and returns a scalar to every element of a DataFrame.

Parameters

funccallable

Python function, returns a single value from a single value.

na_action{None, ‘ignore’}, default None

If ‘ignore’, propagate NaN values, without passing them to func.

New in version 1.2.

**kwargs

Additional keyword arguments to pass as keywords arguments to func.

New in version 1.3.0.

Returns

DataFrame

Transformed DataFrame.

See also

DataFrame.apply

Apply a function along input axis of DataFrame.

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame([[1, 2.12], [3.356, 4.567]])
>>> df
0      1
0  1.000  2.120
1  3.356  4.567
        

Copy

Copied!

            

            
>>> df.applymap(lambda x: len(str(x)))
0  1
0  3  4
1  5  5
        

Like Series.map, NA values can be ignored:

Copy

Copied!

            

            
>>> df_copy = df.copy()
>>> df_copy.iloc[0, 0] = pd.NA
>>> df_copy.applymap(lambda x: len(str(x)), na_action='ignore')
0  1
0  <NA>  4
1     5  5
        

Note that a vectorized version of func often exists, which will be much faster. You could square each number elementwise.

Copy

Copied!

            

            
>>> df.applymap(lambda x: x**2)
0          1
0   1.000000   4.494400
1  11.262736  20.857489
        

But it’s better to avoid applymap in that case.

Copy

Copied!

            

            
>>> df ** 2
0          1
0   1.000000   4.494400
1  11.262736  20.857489
        

asfreq(freq, method=None, how=None, normalize=False, fill_value=None)[source]

Convert time series to specified frequency.

Returns the original data conformed to a new index with the specified frequency.

If the index of this DataFrame is a PeriodIndex, the new index is the result of transforming the original index with PeriodIndex.asfreq (so the original index will map one-to-one to the new index).

Otherwise, the new index will be equivalent to pd.date_range(start, end, freq=freq) where start and end are, respectively, the first and last entries in the original index (see pandas.date_range()). The values corresponding to any timesteps in the new index which were not present in the original index will be null (NaN), unless a method for filling such unknowns is provided (see the method parameter below).

The resample() method is more appropriate if an operation on each group of timesteps (such as an aggregate) is necessary to represent the data at the new frequency.

Parameters

freqDateOffset or str

Frequency DateOffset or string.

method{‘backfill’/’bfill’, ‘pad’/’ffill’}, default None

Method to use for filling holes in reindexed Series (note this does not fill NaNs that already were present):

• ‘pad’ / ‘ffill’: propagate last valid observation forward to next valid

• ‘backfill’ / ‘bfill’: use NEXT valid observation to fill.

how{‘start’, ‘end’}, default end

For PeriodIndex only (see PeriodIndex.asfreq).

normalizebool, default False

Whether to reset output index to midnight.

fill_valuescalar, optional

Value to use for missing values, applied during upsampling (note this does not fill NaNs that already were present).

Returns

DataFrame

DataFrame object reindexed to the specified frequency.

See also

reindex

Conform DataFrame to new index with optional filling logic.

Notes

To learn more about the frequency strings, please see this link.

Examples

Start by creating a series with 4 one minute timestamps.

Copy

Copied!

            

            
>>> index = pd.date_range('1/1/2000', periods=4, freq='T')
>>> series = pd.Series([0.0, None, 2.0, 3.0], index=index)
>>> df = pd.DataFrame({'s': series})
>>> df
s
2000-01-01 00:00:00    0.0
2000-01-01 00:01:00    NaN
2000-01-01 00:02:00    2.0
2000-01-01 00:03:00    3.0
        

Upsample the series into 30 second bins.

Copy

Copied!

            

            
>>> df.asfreq(freq='30S')
s
2000-01-01 00:00:00    0.0
2000-01-01 00:00:30    NaN
2000-01-01 00:01:00    NaN
2000-01-01 00:01:30    NaN
2000-01-01 00:02:00    2.0
2000-01-01 00:02:30    NaN
2000-01-01 00:03:00    3.0
        

Upsample again, providing a fill value.

Copy

Copied!

            

            
>>> df.asfreq(freq='30S', fill_value=9.0)
s
2000-01-01 00:00:00    0.0
2000-01-01 00:00:30    9.0
2000-01-01 00:01:00    NaN
2000-01-01 00:01:30    9.0
2000-01-01 00:02:00    2.0
2000-01-01 00:02:30    9.0
2000-01-01 00:03:00    3.0
        

Upsample again, providing a method.

Copy

Copied!

            

            
>>> df.asfreq(freq='30S', method='bfill')
s
2000-01-01 00:00:00    0.0
2000-01-01 00:00:30    NaN
2000-01-01 00:01:00    NaN
2000-01-01 00:01:30    2.0
2000-01-01 00:02:00    2.0
2000-01-01 00:02:30    3.0
2000-01-01 00:03:00    3.0
        

asof(where, subset=None)[source]

Return the last row(s) without any NaNs before where.

The last row (for each element in where, if list) without any NaN is taken. In case of a DataFrame, the last row without NaN considering only the subset of columns (if not None)

If there is no good value, NaN is returned for a Series or a Series of NaN values for a DataFrame

Parameters

wheredate or array-like of dates

Date(s) before which the last row(s) are returned.

subsetstr or array-like of str, default None

For DataFrame, if not None, only use these columns to check for NaNs.

Returns

scalar, Series, or DataFrame

The return can be:

• scalar : when self is a Series and where is a scalar

• Series: when self is a Series and where is an array-like, or when self is a DataFrame and where is a scalar

• DataFrame : when self is a DataFrame and where is an array-like

Return scalar, Series, or DataFrame.

See also

merge_asof

Perform an asof merge. Similar to left join.

Notes

Dates are assumed to be sorted. Raises if this is not the case.

Examples

A Series and a scalar where.

Copy

Copied!

            

            
>>> s = pd.Series([1, 2, np.nan, 4], index=[10, 20, 30, 40])
>>> s
10    1.0
20    2.0
30    NaN
40    4.0
dtype: float64
        

Copy

Copied!

            

            
>>> s.asof(20)
2.0
        

For a sequence where, a Series is returned. The first value is NaN, because the first element of where is before the first index value.

Copy

Copied!

            

            
>>> s.asof([5, 20])
5     NaN
20    2.0
dtype: float64
        

Missing values are not considered. The following is 2.0, not NaN, even though NaN is at the index location for 30.

Copy

Copied!

            

            
>>> s.asof(30)
2.0
        

Take all columns into consideration

Copy

Copied!

            

            
>>> df = pd.DataFrame({'a': [10, 20, 30, 40, 50],
...                    'b': [None, None, None, None, 500]},
...                   index=pd.DatetimeIndex(['2018-02-27 09:01:00',
...                                           '2018-02-27 09:02:00',
...                                           '2018-02-27 09:03:00',
...                                           '2018-02-27 09:04:00',
...                                           '2018-02-27 09:05:00']))
>>> df.asof(pd.DatetimeIndex(['2018-02-27 09:03:30',
...                           '2018-02-27 09:04:30']))
a   b
2018-02-27 09:03:30 NaN NaN
2018-02-27 09:04:30 NaN NaN
        

Take a single column into consideration

Copy

Copied!

            

            
>>> df.asof(pd.DatetimeIndex(['2018-02-27 09:03:30',
...                           '2018-02-27 09:04:30']),
...         subset=['a'])
a   b
2018-02-27 09:03:30   30.0 NaN
2018-02-27 09:04:30   40.0 NaN
        

assign(**kwargs)[source]

Assign new columns to a DataFrame.

Returns a new object with all original columns in addition to new ones. Existing columns that are re-assigned will be overwritten.

Parameters

**kwargsdict of {str: callable or Series}

The column names are keywords. If the values are callable, they are computed on the DataFrame and assigned to the new columns. The callable must not change input DataFrame (though pandas doesn’t check it). If the values are not callable, (e.g. a Series, scalar, or array), they are simply assigned.

Returns

DataFrame

A new DataFrame with the new columns in addition to all the existing columns.

Notes

Assigning multiple columns within the same assign is possible. Later items in ‘**kwargs’ may refer to newly created or modified columns in ‘df’; items are computed and assigned into ‘df’ in order.

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame({'temp_c': [17.0, 25.0]},
...                   index=['Portland', 'Berkeley'])
>>> df
temp_c
Portland    17.0
Berkeley    25.0
        

Where the value is a callable, evaluated on df:

Copy

Copied!

            

            
>>> df.assign(temp_f=lambda x: x.temp_c * 9 / 5 + 32)
temp_c  temp_f
Portland    17.0    62.6
Berkeley    25.0    77.0
        

Alternatively, the same behavior can be achieved by directly referencing an existing Series or sequence:

Copy

Copied!

            

            
>>> df.assign(temp_f=df['temp_c'] * 9 / 5 + 32)
temp_c  temp_f
Portland    17.0    62.6
Berkeley    25.0    77.0
        

You can create multiple columns within the same assign where one of the columns depends on another one defined within the same assign:

Copy

Copied!

            

            
>>> df.assign(temp_f=lambda x: x['temp_c'] * 9 / 5 + 32,
...           temp_k=lambda x: (x['temp_f'] +  459.67) * 5 / 9)
temp_c  temp_f  temp_k
Portland    17.0    62.6  290.15
Berkeley    25.0    77.0  298.15
        

astype(dtype, copy=True, errors='raise')[source]

Cast a pandas object to a specified dtype dtype.

Parameters

dtypedata type, or dict of column name -> data type

Use a numpy.dtype or Python type to cast entire pandas object to the same type. Alternatively, use {col: dtype, …}, where col is a column label and dtype is a numpy.dtype or Python type to cast one or more of the DataFrame’s columns to column-specific types.

copybool, default True

Return a copy when copy=True (be very careful setting copy=False as changes to values then may propagate to other pandas objects).

errors{‘raise’, ‘ignore’}, default ‘raise’

Control raising of exceptions on invalid data for provided dtype.

• raise : allow exceptions to be raised

• ignore : suppress exceptions. On error return original object.

Returns

castedsame type as caller

See also

to_datetime

Convert argument to datetime.

to_timedelta

Convert argument to timedelta.

to_numeric

Convert argument to a numeric type.

numpy.ndarray.astype

Cast a numpy array to a specified type.

Notes

Deprecated since version 1.3.0:Using astype to convert from timezone-naive dtype to timezone-aware dtype is deprecated and will raise in a future version. Use Series.dt.tz_localize() instead.

Examples

Create a DataFrame:

Copy

Copied!

            

            
>>> d = {'col1': [1, 2], 'col2': [3, 4]}
>>> df = pd.DataFrame(data=d)
>>> df.dtypes
col1    int64
col2    int64
dtype: object
        

Cast all columns to int32:

Copy

Copied!

            

            
>>> df.astype('int32').dtypes
col1    int32
col2    int32
dtype: object
        

Cast col1 to int32 using a dictionary:

Copy

Copied!

            

            
>>> df.astype({'col1': 'int32'}).dtypes
col1    int32
col2    int64
dtype: object
        

Create a series:

Copy

Copied!

            

            
>>> ser = pd.Series([1, 2], dtype='int32')
>>> ser
0    1
1    2
dtype: int32
>>> ser.astype('int64')
0    1
1    2
dtype: int64
        

Convert to categorical type:

Copy

Copied!

            

            
>>> ser.astype('category')
0    1
1    2
dtype: category
Categories (2, int64): [1, 2]
        

Convert to ordered categorical type with custom ordering:

Copy

Copied!

            

            
>>> from pandas.api.types import CategoricalDtype
>>> cat_dtype = CategoricalDtype(
...     categories=[2, 1], ordered=True)
>>> ser.astype(cat_dtype)
0    1
1    2
dtype: category
Categories (2, int64): [2 < 1]
        

Note that using copy=False and changing data on a new pandas object may propagate changes:

Copy

Copied!

            

            
>>> s1 = pd.Series([1, 2])
>>> s2 = s1.astype('int64', copy=False)
>>> s2[0] = 10
>>> s1  # note that s1[0] has changed too
0    10
1     2
dtype: int64
        

Create a series of dates:

Copy

Copied!

            

            
>>> ser_date = pd.Series(pd.date_range('20200101', periods=3))
>>> ser_date
0   2020-01-01
1   2020-01-02
2   2020-01-03
dtype: datetime64[ns]
        

property at: pandas.core.indexing._AtIndexer

Access a single value for a row/column label pair.

Similar to loc, in that both provide label-based lookups. Use at if you only need to get or set a single value in a DataFrame or Series.

Raises

KeyError

If ‘label’ does not exist in DataFrame.

See also

DataFrame.iat

Access a single value for a row/column pair by integer position.

DataFrame.loc

Access a group of rows and columns by label(s).

Series.at

Access a single value using a label.

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame([[0, 2, 3], [0, 4, 1], [10, 20, 30]],
...                   index=[4, 5, 6], columns=['A', 'B', 'C'])
>>> df
A   B   C
4   0   2   3
5   0   4   1
6  10  20  30
        

Get value at specified row/column pair

Copy

Copied!

            

            
>>> df.at[4, 'B']
2
        

Set value at specified row/column pair

Copy

Copied!

            

            
>>> df.at[4, 'B'] = 10
>>> df.at[4, 'B']
10
        

Get value within a Series

Copy

Copied!

            

            
>>> df.loc[5].at['B']
4
        

at_time(time, asof=False, axis=None)[source]

Select values at particular time of day (e.g., 9:30AM).

Parameters

timedatetime.time or str

axis{0 or ‘index’, 1 or ‘columns’}, default 0

Returns

Series or DataFrame

Raises

TypeError

If the index is not a DatetimeIndex

See also

between_time

Select values between particular times of the day.

first

Select initial periods of time series based on a date offset.

last

Select final periods of time series based on a date offset.

DatetimeIndex.indexer_at_time

Get just the index locations for values at particular time of the day.

Examples

Copy

Copied!

            

            
>>> i = pd.date_range('2018-04-09', periods=4, freq='12H')
>>> ts = pd.DataFrame({'A': [1, 2, 3, 4]}, index=i)
>>> ts
A
2018-04-09 00:00:00  1
2018-04-09 12:00:00  2
2018-04-10 00:00:00  3
2018-04-10 12:00:00  4
        

Copy

Copied!

            

            
>>> ts.at_time('12:00')
A
2018-04-09 12:00:00  2
2018-04-10 12:00:00  4
        

property attrs: dict[Hashable, Any]

Dictionary of global attributes of this dataset.

Warning

attrs is experimental and may change without warning.

See also

DataFrame.flags

Global flags applying to this object.

property axes: list[Index]

Return a list representing the axes of the DataFrame.

It has the row axis labels and column axis labels as the only members. They are returned in that order.

Examples

backfill(axis=None, inplace=False, limit=None, downcast=None)[source]

Synonym for DataFrame.fillna() with method='bfill'.

Returns

Series/DataFrame or None

Object with missing values filled or None if inplace=True.

between_time(start_time, end_time, include_start=True, include_end=True, axis=None)[source]

Select values between particular times of the day (e.g., 9:00-9:30 AM).

By setting start_time to be later than end_time, you can get the times that are not between the two times.

Parameters

start_timedatetime.time or str

Initial time as a time filter limit.

end_timedatetime.time or str

End time as a time filter limit.

include_startbool, default True

Whether the start time needs to be included in the result.

include_endbool, default True

Whether the end time needs to be included in the result.

axis{0 or ‘index’, 1 or ‘columns’}, default 0

Determine range time on index or columns value.

Returns

Series or DataFrame

Data from the original object filtered to the specified dates range.

Raises

TypeError

If the index is not a DatetimeIndex

See also

at_time

Select values at a particular time of the day.

first

Select initial periods of time series based on a date offset.

last

Select final periods of time series based on a date offset.

DatetimeIndex.indexer_between_time

Get just the index locations for values between particular times of the day.

Examples

Copy

Copied!

            

            
>>> i = pd.date_range('2018-04-09', periods=4, freq='1D20min')
>>> ts = pd.DataFrame({'A': [1, 2, 3, 4]}, index=i)
>>> ts
A
2018-04-09 00:00:00  1
2018-04-10 00:20:00  2
2018-04-11 00:40:00  3
2018-04-12 01:00:00  4
        

Copy

Copied!

            

            
>>> ts.between_time('0:15', '0:45')
A
2018-04-10 00:20:00  2
2018-04-11 00:40:00  3
        

You get the times that are not between two times by setting start_time later than end_time:

Copy

Copied!

            

            
>>> ts.between_time('0:45', '0:15')
A
2018-04-09 00:00:00  1
2018-04-12 01:00:00  4
        

bfill(axis=None, inplace=False, limit=None, downcast=None)[source]

Synonym for DataFrame.fillna() with method='bfill'.

Returns

Series/DataFrame or None

Object with missing values filled or None if inplace=True.

bool()[source]

Return the bool of a single element Series or DataFrame.

This must be a boolean scalar value, either True or False. It will raise a ValueError if the Series or DataFrame does not have exactly 1 element, or that element is not boolean (integer values 0 and 1 will also raise an exception).

Returns

bool

The value in the Series or DataFrame.

See also

Series.astype

Change the data type of a Series, including to boolean.

DataFrame.astype

Change the data type of a DataFrame, including to boolean.

numpy.bool_

NumPy boolean data type, used by pandas for boolean values.

Examples

The method will only work for single element objects with a boolean value:

Copy

Copied!

            

            
>>> pd.Series([True]).bool()
True
>>> pd.Series([False]).bool()
False
        

Copy

Copied!

            

            
>>> pd.DataFrame({'col': [True]}).bool()
True
>>> pd.DataFrame({'col': [False]}).bool()
False
        

boxplot(column=None, by=None, ax=None, fontsize=None, rot=0, grid=True, figsize=None, layout=None, return_type=None, backend=None, **kwargs)[source]

Make a box plot from DataFrame columns.

Make a box-and-whisker plot from DataFrame columns, optionally grouped by some other columns. A box plot is a method for graphically depicting groups of numerical data through their quartiles. The box extends from the Q1 to Q3 quartile values of the data, with a line at the median (Q2). The whiskers extend from the edges of box to show the range of the data. By default, they extend no more than 1.5 * IQR (IQR = Q3 - Q1) from the edges of the box, ending at the farthest data point within that interval. Outliers are plotted as separate dots.

For further details see Wikipedia’s entry for boxplot.

Parameters

columnstr or list of str, optional

Column name or list of names, or vector. Can be any valid input to pandas.DataFrame.groupby().

bystr or array-like, optional

Column in the DataFrame to pandas.DataFrame.groupby(). One box-plot will be done per value of columns in by.

axobject of class matplotlib.axes.Axes, optional

The matplotlib axes to be used by boxplot.

fontsizefloat or str

Tick label font size in points or as a string (e.g., large).

rotint or float, default 0

The rotation angle of labels (in degrees) with respect to the screen coordinate system.

gridbool, default True

Setting this to True will show the grid.

figsizeA tuple (width, height) in inches

The size of the figure to create in matplotlib.

layouttuple (rows, columns), optional

For example, (3, 5) will display the subplots using 3 columns and 5 rows, starting from the top-left.

return_type{‘axes’, ‘dict’, ‘both’} or None, default ‘axes’

The kind of object to return. The default is axes.

• ‘axes’ returns the matplotlib axes the boxplot is drawn on.

• ‘dict’ returns a dictionary whose values are the matplotlib Lines of the boxplot.

• ‘both’ returns a namedtuple with the axes and dict.

• when grouping with by, a Series mapping columns to return_type is returned.

  If return_type is None, a NumPy array of axes with the same shape as layout is returned.

backendstr, default None

Backend to use instead of the backend specified in the option plotting.backend. For instance, ‘matplotlib’. Alternatively, to specify the plotting.backend for the whole session, set pd.options.plotting.backend.

New in version 1.0.0.

**kwargs

All other plotting keyword arguments to be passed to matplotlib.pyplot.boxplot().

Returns

result

See Notes.

See also

Series.plot.hist

Make a histogram.

matplotlib.pyplot.boxplot

Matplotlib equivalent plot.

Notes

The return type depends on the return_type parameter:

• ‘axes’ : object of class matplotlib.axes.Axes

• ‘dict’ : dict of matplotlib.lines.Line2D objects

• ‘both’ : a namedtuple with structure (ax, lines)

For data grouped with by, return a Series of the above or a numpy array:

• Series

• array (for return_type = None)

Use return_type='dict' when you want to tweak the appearance of the lines after plotting. In this case a dict containing the Lines making up the boxes, caps, fliers, medians, and whiskers is returned.

Examples

Boxplots can be created for every column in the dataframe by df.boxplot() or indicating the columns to be used:

Boxplots of variables distributions grouped by the values of a third variable can be created using the option by. For instance:

A list of strings (i.e. ['X', 'Y']) can be passed to boxplot in order to group the data by combination of the variables in the x-axis:

The layout of boxplot can be adjusted giving a tuple to layout:

Additional formatting can be done to the boxplot, like suppressing the grid (grid=False), rotating the labels in the x-axis (i.e. rot=45) or changing the fontsize (i.e. fontsize=15):

The parameter return_type can be used to select the type of element returned by boxplot. When return_type='axes' is selected, the matplotlib axes on which the boxplot is drawn are returned:

Copy

Copied!

            

            
>>> boxplot = df.boxplot(column=['Col1', 'Col2'], return_type='axes')
>>> type(boxplot)
<class 'matplotlib.axes._subplots.AxesSubplot'>
        

When grouping with by, a Series mapping columns to return_type is returned:

Copy

Copied!

            

            
>>> boxplot = df.boxplot(column=['Col1', 'Col2'], by='X',
...                      return_type='axes')
>>> type(boxplot)
<class 'pandas.core.series.Series'>
        

If return_type is None, a NumPy array of axes with the same shape as layout is returned:

Copy

Copied!

            

            
>>> boxplot = df.boxplot(column=['Col1', 'Col2'], by='X',
...                      return_type=None)
>>> type(boxplot)
<class 'numpy.ndarray'>
        

clip(lower=None, upper=None, axis=None, inplace=False, *args, **kwargs)[source]

Trim values at input threshold(s).

Assigns values outside boundary to boundary values. Thresholds can be singular values or array like, and in the latter case the clipping is performed element-wise in the specified axis.

Parameters

lowerfloat or array-like, default None

Minimum threshold value. All values below this threshold will be set to it. A missing threshold (e.g NA) will not clip the value.

upperfloat or array-like, default None

Maximum threshold value. All values above this threshold will be set to it. A missing threshold (e.g NA) will not clip the value.

axisint or str axis name, optional

Align object with lower and upper along the given axis.

inplacebool, default False

Whether to perform the operation in place on the data.

*args, **kwargs

Additional keywords have no effect but might be accepted for compatibility with numpy.

Returns

Series or DataFrame or None

Same type as calling object with the values outside the clip boundaries replaced or None if inplace=True.

See also

Series.clip

Trim values at input threshold in series.

DataFrame.clip

Trim values at input threshold in dataframe.

numpy.clip

Clip (limit) the values in an array.

Examples

Copy

Copied!

            

            
>>> data = {'col_0': [9, -3, 0, -1, 5], 'col_1': [-2, -7, 6, 8, -5]}
>>> df = pd.DataFrame(data)
>>> df
col_0  col_1
0      9     -2
1     -3     -7
2      0      6
3     -1      8
4      5     -5
        

Clips per column using lower and upper thresholds:

Copy

Copied!

            

            
>>> df.clip(-4, 6)
col_0  col_1
0      6     -2
1     -3     -4
2      0      6
3     -1      6
4      5     -4
        

Clips using specific lower and upper thresholds per column element:

Copy

Copied!

            

            
>>> t = pd.Series([2, -4, -1, 6, 3])
>>> t
0    2
1   -4
2   -1
3    6
4    3
dtype: int64
        

Copy

Copied!

            

            
>>> df.clip(t, t + 4, axis=0)
col_0  col_1
0      6      2
1     -3     -4
2      0      3
3      6      8
4      5      3
        

Clips using specific lower threshold per column element, with missing values:

Copy

Copied!

            

            
>>> t = pd.Series([2, -4, np.NaN, 6, 3])
>>> t
0    2.0
1   -4.0
2    NaN
3    6.0
4    3.0
dtype: float64
        

Copy

Copied!

            

            
>>> df.clip(t, axis=0)
col_0  col_1
0      9      2
1     -3     -4
2      0      6
3      6      8
4      5      3
        

columns: Index

The column labels of the DataFrame.

combine(other, func, fill_value=None, overwrite=True)[source]

Perform column-wise combine with another DataFrame.

Combines a DataFrame with other DataFrame using func to element-wise combine columns. The row and column indexes of the resulting DataFrame will be the union of the two.

Parameters

otherDataFrame

The DataFrame to merge column-wise.

funcfunction

Function that takes two series as inputs and return a Series or a scalar. Used to merge the two dataframes column by columns.

fill_valuescalar value, default None

The value to fill NaNs with prior to passing any column to the merge func.

overwritebool, default True

If True, columns in self that do not exist in other will be overwritten with NaNs.

Returns

DataFrame

Combination of the provided DataFrames.

See also

DataFrame.combine_first

Combine two DataFrame objects and default to non-null values in frame calling the method.

Examples

Combine using a simple function that chooses the smaller column.

Copy

Copied!

            

            
>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> take_smaller = lambda s1, s2: s1 if s1.sum() < s2.sum() else s2
>>> df1.combine(df2, take_smaller)
A  B
0  0  3
1  0  3
        

Example using a true element-wise combine function.

Copy

Copied!

            

            
>>> df1 = pd.DataFrame({'A': [5, 0], 'B': [2, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> df1.combine(df2, np.minimum)
A  B
0  1  2
1  0  3
        

Using fill_value fills Nones prior to passing the column to the merge function.

Copy

Copied!

            

            
>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> df1.combine(df2, take_smaller, fill_value=-5)
A    B
0  0 -5.0
1  0  4.0
        

However, if the same element in both dataframes is None, that None is preserved

Copy

Copied!

            

            
>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [None, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [None, 3]})
>>> df1.combine(df2, take_smaller, fill_value=-5)
A    B
0  0 -5.0
1  0  3.0
        

Example that demonstrates the use of overwrite and behavior when the axis differ between the dataframes.

Copy

Copied!

            

            
>>> df1 = pd.DataFrame({'A': [0, 0], 'B': [4, 4]})
>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [-10, 1], }, index=[1, 2])
>>> df1.combine(df2, take_smaller)
A    B     C
0  NaN  NaN   NaN
1  NaN  3.0 -10.0
2  NaN  3.0   1.0
        

Copy

Copied!

            

            
>>> df1.combine(df2, take_smaller, overwrite=False)
A    B     C
0  0.0  NaN   NaN
1  0.0  3.0 -10.0
2  NaN  3.0   1.0
        

Demonstrating the preference of the passed in dataframe.

Copy

Copied!

            

            
>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1], }, index=[1, 2])
>>> df2.combine(df1, take_smaller)
A    B   C
0  0.0  NaN NaN
1  0.0  3.0 NaN
2  NaN  3.0 NaN
        

Copy

Copied!

            

            
>>> df2.combine(df1, take_smaller, overwrite=False)
A    B   C
0  0.0  NaN NaN
1  0.0  3.0 1.0
2  NaN  3.0 1.0
        

combine_first(other)[source]

Update null elements with value in the same location in other.

Combine two DataFrame objects by filling null values in one DataFrame with non-null values from other DataFrame. The row and column indexes of the resulting DataFrame will be the union of the two.

Parameters

otherDataFrame

Provided DataFrame to use to fill null values.

Returns

DataFrame

The result of combining the provided DataFrame with the other object.

See also

DataFrame.combine

Perform series-wise operation on two DataFrames using a given function.

Examples

Copy

Copied!

            

            
>>> df1 = pd.DataFrame({'A': [None, 0], 'B': [None, 4]})
>>> df2 = pd.DataFrame({'A': [1, 1], 'B': [3, 3]})
>>> df1.combine_first(df2)
A    B
0  1.0  3.0
1  0.0  4.0
        

Null values still persist if the location of that null value does not exist in other

Copy

Copied!

            

            
>>> df1 = pd.DataFrame({'A': [None, 0], 'B': [4, None]})
>>> df2 = pd.DataFrame({'B': [3, 3], 'C': [1, 1]}, index=[1, 2])
>>> df1.combine_first(df2)
A    B    C
0  NaN  4.0  NaN
1  0.0  3.0  1.0
2  NaN  3.0  1.0
        

compare(other, align_axis=1, keep_shape=False, keep_equal=False)[source]

Compare to another DataFrame and show the differences.

New in version 1.1.0.

Parameters

otherDataFrame

Object to compare with.

align_axis{0 or ‘index’, 1 or ‘columns’}, default 1

Determine which axis to align the comparison on.

• 0, or ‘index’Resulting differences are stacked vertically

  with rows drawn alternately from self and other.

• 1, or ‘columns’Resulting differences are aligned horizontally

  with columns drawn alternately from self and other.

keep_shapebool, default False

If true, all rows and columns are kept. Otherwise, only the ones with different values are kept.

keep_equalbool, default False

If true, the result keeps values that are equal. Otherwise, equal values are shown as NaNs.

Returns

DataFrame

DataFrame that shows the differences stacked side by side.

The resulting index will be a MultiIndex with ‘self’ and ‘other’ stacked alternately at the inner level.

Raises

ValueError

When the two DataFrames don’t have identical labels or shape.

See also

Series.compare

Compare with another Series and show differences.

DataFrame.equals

Test whether two objects contain the same elements.

Notes

Matching NaNs will not appear as a difference.

Can only compare identically-labeled (i.e. same shape, identical row and column labels) DataFrames

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame(
...     {
...         "col1": ["a", "a", "b", "b", "a"],
...         "col2": [1.0, 2.0, 3.0, np.nan, 5.0],
...         "col3": [1.0, 2.0, 3.0, 4.0, 5.0]
...     },
...     columns=["col1", "col2", "col3"],
... )
>>> df
col1  col2  col3
0    a   1.0   1.0
1    a   2.0   2.0
2    b   3.0   3.0
3    b   NaN   4.0
4    a   5.0   5.0
        

Copy

Copied!

            

            
>>> df2 = df.copy()
>>> df2.loc[0, 'col1'] = 'c'
>>> df2.loc[2, 'col3'] = 4.0
>>> df2
col1  col2  col3
0    c   1.0   1.0
1    a   2.0   2.0
2    b   3.0   4.0
3    b   NaN   4.0
4    a   5.0   5.0
        

Align the differences on columns

Copy

Copied!

            

            
>>> df.compare(df2)
col1       col3
self other self other
0    a     c  NaN   NaN
2  NaN   NaN  3.0   4.0
        

Stack the differences on rows

Copy

Copied!

            

            
>>> df.compare(df2, align_axis=0)
col1  col3
0 self     a   NaN
other    c   NaN
2 self   NaN   3.0
other  NaN   4.0
        

Keep the equal values

Copy

Copied!

            

            
>>> df.compare(df2, keep_equal=True)
col1       col3
self other self other
0    a     c  1.0   1.0
2    b     b  3.0   4.0
        

Keep all original rows and columns

Copy

Copied!

            

            
>>> df.compare(df2, keep_shape=True)
col1       col2       col3
self other self other self other
0    a     c  NaN   NaN  NaN   NaN
1  NaN   NaN  NaN   NaN  NaN   NaN
2  NaN   NaN  NaN   NaN  3.0   4.0
3  NaN   NaN  NaN   NaN  NaN   NaN
4  NaN   NaN  NaN   NaN  NaN   NaN
        

Keep all original rows and columns and also all original values

Copy

Copied!

            

            
>>> df.compare(df2, keep_shape=True, keep_equal=True)
col1       col2       col3
self other self other self other
0    a     c  1.0   1.0  1.0   1.0
1    a     a  2.0   2.0  2.0   2.0
2    b     b  3.0   3.0  3.0   4.0
3    b     b  NaN   NaN  4.0   4.0
4    a     a  5.0   5.0  5.0   5.0
        

convert_dtypes(infer_objects=True, convert_string=True, convert_integer=True, convert_boolean=True, convert_floating=True)[source]

Convert columns to best possible dtypes using dtypes supporting pd.NA.

New in version 1.0.0.

Parameters

infer_objectsbool, default True

Whether object dtypes should be converted to the best possible types.

convert_stringbool, default True

Whether object dtypes should be converted to StringDtype().

convert_integerbool, default True

Whether, if possible, conversion can be done to integer extension types.

convert_booleanbool, defaults True

Whether object dtypes should be converted to BooleanDtypes().

convert_floatingbool, defaults True

Whether, if possible, conversion can be done to floating extension types. If convert_integer is also True, preference will be give to integer dtypes if the floats can be faithfully casted to integers.

New in version 1.2.0.

Returns

Series or DataFrame

Copy of input object with new dtype.

See also

infer_objects

Infer dtypes of objects.

to_datetime

Convert argument to datetime.

to_timedelta

Convert argument to timedelta.

to_numeric

Convert argument to a numeric type.

Notes

By default, convert_dtypes will attempt to convert a Series (or each Series in a DataFrame) to dtypes that support pd.NA. By using the options convert_string, convert_integer, convert_boolean and convert_boolean, it is possible to turn off individual conversions to StringDtype, the integer extension types, BooleanDtype or floating extension types, respectively.

For object-dtyped columns, if infer_objects is True, use the inference rules as during normal Series/DataFrame construction. Then, if possible, convert to StringDtype, BooleanDtype or an appropriate integer or floating extension type, otherwise leave as object.

If the dtype is integer, convert to an appropriate integer extension type.

If the dtype is numeric, and consists of all integers, convert to an appropriate integer extension type. Otherwise, convert to an appropriate floating extension type.

Changed in version 1.2:Starting with pandas 1.2, this method also converts float columns to the nullable floating extension type.

In the future, as new dtypes are added that support pd.NA, the results of this method will change to support those new dtypes.

Examples

Copy

Copied!

            

            
>>> df = pd.DataFrame(
...     {
...         "a": pd.Series([1, 2, 3], dtype=np.dtype("int32")),
...         "b": pd.Series(["x", "y", "z"], dtype=np.dtype("O")),
...         "c": pd.Series([True, False, np.nan], dtype=np.dtype("O")),
...         "d": pd.Series(["h", "i", np.nan], dtype=np.dtype("O")),
...         "e": pd.Series([10, np.nan, 20], dtype=np.dtype("float")),
...         "f": pd.Series([np.nan, 100.5, 200], dtype=np.dtype("float")),
...     }
... )
        

Start with a DataFrame with default dtypes.

Copy

Copied!

            

            
>>> df
a  b      c    d     e      f
0  1  x   True    h  10.0    NaN
1  2  y  False    i   NaN  100.5
2  3  z    NaN  NaN  20.0  200.0
        

Copy

Copied!

            

            
>>> df.dtypes
a      int32
b     object
c     object
d     object
e    float64
f    float64
dtype: object
        

Convert the DataFrame to use best possible dtypes.

Copy

Copied!

            

            
>>> dfn = df.convert_dtypes()
>>> dfn
a  b      c     d     e      f
0  1  x   True     h    10   <NA>
1  2  y  False     i  <NA>  100.5
2  3  z   <NA>  <NA>    20  200.0
        

Copy

Copied!

            

            
>>> dfn.dtypes
a      Int32
b     string
c    boolean
d     string
e      Int64
f    Float64
dtype: object
        

Start with a Series of strings and missing data represented by np.nan.

Copy

Copied!

            

            
>>> s = pd.Series(["a", "b", np.nan])
>>> s
0      a
1      b
2    NaN
dtype: object
        

Obtain a Series with dtype StringDtype.

Copy

Copied!

            

            
>>> s.convert_dtypes()
0       a
1       b
2    <NA>
dtype: string