Please note: This course was previously offered under the title “Big Data, Introduction.”
In the era of big data and compute-intensive analytics, the ability to write high-performance Python code is essential. This course is designed for learners with basic Python knowledge who want to handle large volumes of data efficiently and optimize their workflows. We will explore how to make Python performant—moving beyond basic pandas use—by introducing tools, techniques, and tradeoffs for improving execution speed, memory use, and scalability.
You will learn strategies such as vectorization, avoiding unnecessary loops, leveraging data structures like NumPy arrays, and using multithreading/multiprocessing. We will also explore distributed computing with PySpark and Dask, and introduce Polars as a cutting-edge alternative to pandas. These skills will be placed in the broader context of big data frameworks and architectures, including Apache Spark, Apache Kafka, and modern NoSQL databases like MongoDB and Cassandra. GPU optimization techniques will also be discussed at an introductory level.
The final project will integrate these concepts into the design of a high-performance data processing pipeline, giving you hands-on experience with tools and methods to analyze large datasets efficiently.
Basic Python programming knowledge and familiarity with Python data analysis libraries such as pandas, or completion of a course such as DBDA.X420 - “Python for Data Analysis.”
10/22/25: Instructor TBA.
This class meets simultaneously in a classroom and remotely via Zoom. Students are expected to attend and participate in the course, either in-person or remotely, during the days and times that are specified on the course schedule. Students attending remotely are also strongly encouraged to have their cameras on to get the most out of the remote learning experience. Students attending the class in-person are expected to bring a laptop to each class meeting.
To see all meeting dates, click "Full Schedule" below.
You will be granted access in Canvas to your course site and course materials approximately 24 hours prior to the published start date of the course.
