Python and Spark for Big Data in Marina Bay, Singapore

This course provides a comprehensive introduction to Big Data processing using Python and Apache Spark. It covers the fundamentals of distributed computing, working with PySpark, data ingestion, transformations, and optimizations for handling large-scale datasets. Participants will gain hands-on experience in leveraging Spark for real-world data analytics and machine learning applications.

Course Agenda

Introduction to Big Data and Apache Spark

What is Big Data? Challenges and opportunities
Introduction to Apache Spark and its ecosystem
Comparison of Spark with Hadoop MapReduce
Setting up Spark with Python (PySpark)

Spark Architecture and Execution Model

Spark components: Driver, Executors, Cluster Manager
RDDs (Resilient Distributed Datasets) vs. DataFrames vs. Datasets
Spark execution flow and DAG (Directed Acyclic Graph)

Working with Spark RDDs and Transformations

Creating and manipulating RDDs
Lazy evaluation and transformations (map, filter, reduceByKey)
Actions (collect, count, take, foreach)

Spark DataFrames and Spark SQL

Introduction to DataFrames and their advantages
Schema inference and working with structured data
Performing SQL operations using Spark SQL
Reading and writing data from various sources (CSV, JSON, Parquet)

Data Processing with PySpark

Handling missing data and data cleansing techniques
Aggregation and group operations
Joins and window functions in PySpark

Optimizing Spark Performance

Partitioning strategies and shuffling
Caching and persistence
Broadcast variables and accumulator optimization
Tuning Spark jobs for better resource utilization

Machine Learning with Spark MLlib

Introduction to MLlib and Spark ML Pipelines
Building and evaluating machine learning models
Feature engineering with PySpark
Deploying machine learning models in Spark

Real-World Big Data Use Cases

Processing large datasets with Spark
ETL pipelines using PySpark
Streaming data processing with Spark Streaming
Integration with cloud platforms (AWS, Azure, GCP)

Hands-on Project and Capstone Assignment

Developing an end-to-end Big Data processing pipeline
Implementing Spark jobs for large-scale data analytics
Optimizing performance for scalability