PySpark for Data Science - I: Fundamentals

Dive into the world of big data processing with PySpark, the Python library for Apache Spark. Learn how to process, analyze, and derive insights from massive datasets using Python’s user-friendly interface. Elevate your data skills with PySpark. Dive deep into distributed data processing, machine learning, streaming, and more to navigate the vast oceans of big data.

Learn how to process, analyze, and derive insights from massive datasets using Python’s user-friendly interface.
Elevate your data skills with PySpark. Dive deep into distributed data processing, machine learning, streaming, and more to navigate the vast oceans of big data.

Prabhakaran SPrincipal Data Scientist

Join All-Access Watch 1 free lesson

Included in All-Access, one membership for every course.

Preview the first lesson · 4:34

Lessons: 25, in 5 chapters
Video: 2 h 3 min
Free to watch: 1 lesson
Certificate: On completion

Course content

5 chapters · 25 lessons · 2 h 3 min · 1 free to preview

01Sec 1: Introduction to PySpark4 lessons · 24 min
1. What is PySparkPreview4:34
2. Download Resources1:00
3. Why every Data Engineer and Data Scientist should learn PySpark?12:09
4. How PySpark works?6:06
02Sec 2: The Spark session and Spark Dataframes5 lessons · 38 min
1. What is Spark session and How to create one?16:36
2. Configuring a SparkSession4:47
3. How to create PySpark Dataframe from various sources?12:12
4. How to use SQL on PySpark Dataframe?3:06
5. Course Review1:00
03Sec 3: PySpark Data Wrangling Techniques5 lessons · 16 min
1. Selecting Columns5:08
2. Dropping Columns0:52
3. Renaming Columns2:22
4. Select and Filter rows2:12
5. How to sort PySpark DataFrame5:01
04Sec 4: Aggregation and custom methods5 lessons · 34 min
1. Data Aggregation5:10
2. Extending Spark using UDF’s16:53
3. Exercise PySpark UDF1:21
4. Pandas UDF Custom Functions8:21
5. Pandas UDAF User defined aggregation function2:42
05Sec 5: Joins and Pivoting6 lessons · 13 min
1. PySpark Joins Part 14:51
2. PySpark Joins Part 21:49
3. How to define Schema of Dataframe?1:43
4. PySpark Union1:02
5. Pivoting1:51
6. How to Unpivot PySpark Dataframe2:01

What you need

Basics of Python
Foundational knowledge of Data Science
High school maths

Who this course is for

Data Science Aspirants
Data Science Professionals
Software/Data engineers interested in quantitative analysis
Professionals working with large datasets
Data analysts, economists, researchers

Your instructor

Prabhakaran S

Principal Data Scientist

My name is Selva, and I am super excited to mentor you on this course!

I head the Data Science team for a global Fortune 500 company and over the last 10 years of my data science experience I've deployed 20+ global products. I'm also the Founder and Chief Author of Machine Learning Plus, which has over 4M annual readers.

I specialize in covering the in-depth intuition and maths of any concept or algorithm. And based on my existing student requests, I've put up the series of courses and projects with detailed explanations, just like on-the-job experience. Hope you love it!

from 3,965 ratings: 4.92
learners: 57K+
courses: 85

Questions

How do I get this course?

It is part of All-Access: one membership that opens all 85 courses. The plans and prices are on our course site. You can watch the free preview lessons before you join.

Can I get a refund?

Yes. If the membership is not right for you, ask for a refund within 30 days of your first payment.

Can I learn on my phone?

Yes. Lessons play in the browser on any phone, tablet or computer, and your place is saved for the next device.

I am already a member. Where are my courses?

On our course site, edu.machinelearningplus.com. Sign in there with the email you joined with. The courses move to this site later, with your access, and you will not pay again.