E6893 Big Data Analytics Lecture 5: End-to-End System Workflow Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science October 6th, 2023 E6893 Big Data Analytics — Lecture 5 © CY Lin, 2023 Columbia University Workflow 2 © CY Lin, Columbia University E6893 Big Data Analytics — Lecture 5 fl fl A scheduler, which handles both triggering scheduled work ows, and submitting Tasks to the executor to run. An executor, which handles running tasks. In the default Air ow installation, this runs everything inside the scheduler, but most production-suitable executors actually push task execution out to workers. A webserver, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks. A folder of DAG les, read by the scheduler and executor (and any workers the executor has) A metadata database, used by the scheduler, executor and webserver to store state. fi Workflow Components 3 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University Apache Airflow 4 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University Workloads There are three types of Tasks: Operators, prede ned tasks that you can string together quickly to build most parts of your DAGs. Sensors, a special subclass of Operators which are entirely about waiting for an external event to happen. A TaskFlow-decorated @task, which is a custom Python function packaged up as a Task. E6893 Big Data Analytics — Lecture 5 fi 5 © CY Lin, Columbia University Operators 6 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University Example of Popular Operators 7 BashOperator - executes a bash command PythonOperator - calls an arbitrary Python function EmailOperator - sends an email SimpleHttpOperator MySqlOperator PostgresOperator MsSqlOperator OracleOperator JdbcOperator DockerOperator HiveOperator S3FileTransformOperator PrestoToMySqlOperator Slack