pyspark.pandas.DataFrame.shift

DataFrame.shift(periods: int = 1, fill_value: Optional[Any] = None) → pyspark.pandas.frame.DataFrame

Shift DataFrame by desired number of periods.

Note

the current implementation of shift uses Spark’s Window without specifying partition specification. This leads to move all data into single partition in single machine and could cause serious performance degradation. Avoid this method against very large dataset.

Parameters

periodsint

Number of periods to shift. Can be positive or negative.

fill_valueobject, optional

The scalar value to use for newly introduced missing values. The default depends on the dtype of self. For numeric data, np.nan is used.

Returns

Copy of input DataFrame, shifted.

Examples

>>> df = ps.DataFrame({'Col1': [10, 20, 15, 30, 45],
...                    'Col2': [13, 23, 18, 33, 48],
...                    'Col3': [17, 27, 22, 37, 52]},
...                   columns=['Col1', 'Col2', 'Col3'])

>>> df.shift(periods=3)
   Col1  Col2  Col3
0   NaN   NaN   NaN
1   NaN   NaN   NaN
2   NaN   NaN   NaN
3  10.0  13.0  17.0
4  20.0  23.0  27.0

>>> df.shift(periods=3, fill_value=0)
   Col1  Col2  Col3
0     0     0     0
1     0     0     0
2     0     0     0
3    10    13    17
4    20    23    27

pyspark.pandas.DataFrame.insert pyspark.pandas.DataFrame.first_valid_index