PySpark for Data Science - II: Statistics for Big Data

Dive into the world of big data processing with PySpark, the Python library for Apache Spark. Learn how to process, analyze, and derive insights from massive datasets using Python’s user-friendly interface. Elevate your data skills with PySpark. Dive deep into distributed data processing, machine learning, streaming, and more to navigate the vast oceans of big data.

  • Learn how to process, analyze, and derive insights from massive datasets using Python’s user-friendly interface.
  • Elevate your data skills with PySpark. Dive deep into distributed data processing, machine learning, streaming, and more to navigate the vast oceans of big data.

Prabhakaran SPrincipal Data Scientist

Join All-AccessWatch 1 free lesson

Included in All-Access, one membership for every course.

Preview the first lesson · 6:07
Lessons
23, in 11 chapters
Video
1 h 20 min
Free to watch
1 lesson
Certificate
On completion

Course content

11 chapters · 23 lessons · 1 h 20 min · 1 free to preview

  1. 01Introduction to Statistics for Big Data2 lessons · 7 min
    1. Introduction to Statistics for Big DataPreview6:07
    2. Download Resources1:00
  2. 02Mean2 lessons · 11 min
    1. Calculate the mean of a list6:41
    2. Mean for a PySpark DataFrame4:23
  3. 03Median4 lessons · 13 min
    1. How to calculate the Median of a list?5:32
    2. Calculate Median of a PySpark DataFrame column1:28
    3. Using PySpark Window function5:26
    4. Course Review1:00
  4. 04Mode2 lessons · 9 min
    1. How to calculate the Mode of a list7:12
    2. How to calculate Mode on a PySpark DataFrame?1:51
  5. 05Deciles and Quartiles2 lessons · 5 min
    1. How to calculate deciles and quantiles?4:14
    2. Using approxQuantile on multiple columns0:38
  6. 06Variance2 lessons · 8 min
    1. Calculating Using PySpark RDD's5:15
    2. On PySpark DataFrame columns2:35
  7. 07Standard Deviation3 lessons · 3 min
    1. Using the describe() function0:46
    2. Using the agg() based methods1:17
    3. Using the selectExpr() function with SQL expressions0:36
  8. 08Correlation3 lessons · 7 min
    1. Using DataFrame API2:57
    2. Using MLlib2:10
    3. Correlation matrix and heat map2:19
  9. 09T-Test1 lesson · 6 min
    1. T-Test5:42
  10. 10F-Test1 lesson · 6 min
    1. F-Test5:42
  11. 11Chi-Square Test1 lesson · 7 min
    1. Chi-Square Test7:01

What you need

  • Basics of Python
  • Foundational knowledge of Data Science
  • High school maths

Who this course is for

  • Data Science Aspirants
  • Data Science Professionals
  • Software/Data engineers interested in quantitative analysis
  • Professionals working with large datasets
  • Data analysts, economists, researchers

Your instructor

Prabhakaran S

Prabhakaran S

Principal Data Scientist

My name is Selva, and I am super excited to mentor you on this course!

I head the Data Science team for a global Fortune 500 company and over the last 10 years of my data science experience I've deployed 20+ global products. I'm also the Founder and Chief Author of Machine Learning Plus, which has over 4M annual readers.

I specialize in covering the in-depth intuition and maths of any concept or algorithm. And based on my existing student requests, I've put up the series of courses and projects with detailed explanations, just like on-the-job experience. Hope you love it!

from 3,965 ratings
4.92
learners
57K+
courses
85

Questions

How do I get this course?

It is part of All-Access: one membership that opens all 85 courses. The plans and prices are on our course site. You can watch the free preview lessons before you join.

Can I get a refund?

Yes. If the membership is not right for you, ask for a refund within 30 days of your first payment.

Can I learn on my phone?

Yes. Lessons play in the browser on any phone, tablet or computer, and your place is saved for the next device.

I am already a member. Where are my courses?

On our course site, edu.machinelearningplus.com. Sign in there with the email you joined with. The courses move to this site later, with your access, and you will not pay again.

More in MLOps and Deployment Specialization

See the whole specialization
PySpark for Data Science - III: Data Cleaning and Analysis MethodsDive into the world of big data processing with PySpark, the Python library for Apache Spark. Learn how to process, analyze, and derive insights from massive datasets using Python’s user-friendly interface. Elevate your data skills with PySpark. Dive deep into distributed data processing, machine learning, streaming, and more to navigate the vast oceans of big data.22 lessons1 h 14 minCertificateIn All-AccessView coursePySpark for Data Science - IV: Machine LearningDive into the world of big data processing with PySpark, the Python library for Apache Spark. Learn how to process, analyze, and derive insights from massive datasets using Python’s user-friendly interface. Elevate your data skills with PySpark. Dive deep into distributed data processing, machine learning, streaming, and more to navigate the vast oceans of big data.27 lessons1 h 19 minCertificateIn All-AccessView coursePySpark for Data Science - V: ML PipelinesDive into the world of big data processing with PySpark, the Python library for Apache Spark. Learn how to process, analyze, and derive insights from massive datasets using Python’s user-friendly interface. Elevate your data skills with PySpark. Dive deep into distributed data processing, machine learning, streaming, and more to navigate the vast oceans of big data.18 lessons1 h 2 minCertificateIn All-AccessView course
Contact us