Migration Guide

This page describes the migration guide specific to PySpark.

• Upgrading from PySpark 3.1 to 3.2
• Upgrading from PySpark 2.4 to 3.0
• Upgrading from PySpark 2.3 to 2.4
• Upgrading from PySpark 2.3.0 to 2.3.1 and above
• Upgrading from PySpark 2.2 to 2.3
• Upgrading from PySpark 1.4 to 1.5
• Upgrading from PySpark 1.0-1.2 to 1.3

The guide below is for those who are from Koalas.

• Migrating from Koalas to pandas API on Spark

Many items of other migration guides can also be applied when migrating PySpark to higher versions because PySpark internally shares other components. Please also refer other migration guides:

• Migration Guide: Spark Core

• Migration Guide: SQL, Datasets and DataFrame

• Migration Guide: Structured Streaming

• Migration Guide: MLlib (Machine Learning)

previous

Setting up IDEs

next

Upgrading from PySpark 3.1 to 3.2