Read and Write files using PySpark – Multiple ways to Read and Write data using PySpark
Introduction
Apache PySpark is an open-source, distributed computing system designed for big data processing and analytics. It provides an interface for programming Apache Spark with the Python programming language.
One of the most important tasks in data processing is reading and writing data to various file formats. In this blog post, we will explore multiple ways to read and write data using PySpark with code examples.
1. Prerequisites
To follow this tutorial, you’ll need to have PySpark installed on your system. You can install PySpark using pip:
pip install pyspark
pip install findspark
2. Initializing Spark Session
Before we dive into reading and writing data, let’s initialize a SparkSession. The SparkSession is the entry point to PySpark and allows you to interact with the data
import findspark
findspark.init()
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("Read and Write Data Using PySpark") \
.getOrCreate()
3. Create a PySpark DataFrame
Creating a DataFrame from a Python list of dictionaries:
data = [("Alice", 34), ("Bob", 45), ("Cathy", 29)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)
df.show()
+-----+---+
| Name|Age|
+-----+---+
|Alice| 34|
| Bob| 45|
|Cathy| 29|
+-----+---+
4. Reading and Writing CSV Files
To read a CSV file using PySpark, you can use the read.csv() method:
csv_file = "path/to/your/csv/file.csv"
df_csv = spark.read.csv(csv_file, header=True, inferSchema=True)
Now that you have your data in a DataFrame, you can write it back to a CSV file using the write.csv() method:
output_path = "path/to/output/csv/file.csv"
df_csv.write.csv(output_path, header=True, mode="overwrite")
5. Reading and Writing JSON Files
To read a JSON file using PySpark, you can use the read.json() method:
json_file = "path/to/your/json/file.json"
df_json = spark.read.json(json_file)
You can write the data back to a JSON file using the write.json() method:
output_path = "path/to/output/json/file.json"
df_json.write.json(output_path, mode="overwrite")
6. Reading and Writing Parquet Files
To read a Parquet file using PySpark, you can use the read.parquet() method:
parquet_file = "path/to/your/parquet/file.parquet"
df_parquet = spark.read.parquet(parquet_file)
To write the data back to a Parquet file, use the write.parquet() method:
output_path = "path/to/output/parquet/file.parquet"
df_parquet.write.parquet(output_path, mode="overwrite")
7. Creating a SQL Table in PySpark
We’ll create a sample DataFrame using a list of dictionaries and register the DataFrame as a temporary SQL table to perform SQL operations
data = [
{"name": "Alice", "age": 30, "city": "New York"},
{"name": "Bob", "age": 25, "city": "San Francisco"},
{"name": "Charlie", "age": 35, "city": "Los Angeles"}
]
df = spark.createDataFrame(data)
df.createOrReplaceTempView("people")
query = "SELECT * FROM people WHERE age >= 30"
result_df = spark.sql(query)
result_df.show()
+---+-----------+-------+
|age| city| name|
+---+-----------+-------+
| 30| New York| Alice|
| 35|Los Angeles|Charlie|
+---+-----------+-------+
8. Convert Pandas Dataframe to PySpark Dataframe
import pandas as pd
data = [
{"name": "Alice", "age": 30, "city": "New York"},
{"name": "Bob", "age": 25, "city": "San Francisco"},
{"name": "Charlie", "age": 35, "city": "Los Angeles"}
]
pandasDF = pd.DataFrame(data, columns = ['name', 'age', 'city'])
#Create PySpark DataFrame from Pandas
sparkDF=spark.createDataFrame(pandasDF)
sparkDF.printSchema()
sparkDF.show()
root
|-- name: string (nullable = true)
|-- age: long (nullable = true)
|-- city: string (nullable = true)
+-------+---+-------------+
| name|age| city|
+-------+---+-------------+
| Alice| 30| New York|
| Bob| 25|San Francisco|
|Charlie| 35| Los Angeles|
+-------+---+-------------+
Conclusion
In this blog post, we explored multiple ways to read and write data using PySpark, including CSV, JSON, Parquet, SQL, Pandas Data Frame
Build a strong Python foundation with hands-on exercises designed for aspiring Data Scientists and AI/ML Engineers.
Start Free Course →
Related Articles
