E6893 Big Data Analytics Lecture 5: End-to-End System Workflow Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science October 6th, 2023 E6893 Big Data Analytics — Lecture 5 © CY Lin, 2023 Columbia University
Workflow • A scheduler, which handles both triggering scheduled workflows, and submitting Tasks to the executor to run. • An executor, which handles running tasks. In the default Airflow installation, this runs everything inside the scheduler, but most production-suitable executors actually push task execution out to workers. • A webserver, which presents a handy user interface to inspect, trigger and debug the behaviour of DAGs and tasks. • A folder of DAG files, read by the scheduler and executor (and any workers the executor has) • A metadata database, used by the scheduler, executor and webserver to store state. 2 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University
Workloads There are three types of Tasks: • Operators, predefined tasks that you can string together quickly to build most parts of your DAGs. • Sensors, a special subclass of Operators which are entirely about waiting for an external event to happen. • A TaskFlow-decorated @task, which is a custom Python function packaged up as a Task. 5 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University
Example of Popular Operators • BashOperator - executes a bash command • PythonOperator - calls an arbitrary Python function • EmailOperator - sends an email • SimpleHttpOperator • MySqlOperator • PostgresOperator • MsSqlOperator • OracleOperator • JdbcOperator • DockerOperator • HiveOperator • S3FileTransformOperator • PrestoToMySqlOperator • SlackAPIOperator 7 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University
Sensors Sensors are a special type of Operator that are designed to do exactly one thing - wait for something to occur. It can be time-based, or waiting for a file, or an external event, but all they do is wait until something happens, and then succeed so their downstream tasks can run. Because they are primarily idle, Sensors have three different modes of running so you can be a bit more efficient about using them: • poke (default): The Sensor takes up a worker slot for its entire runtime • reschedule: The Sensor takes up a worker slot only when it is checking, and sleeps for a set duration between checks • smart sensor: There is a single centralized version of this Sensor that batches all executions of it 8 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University
TaskFlow askFlow takes care of moving inputs and outputs between your Tasks as well as automatically calculating dependencies - when you call a TaskFlow function in your DAG file, rather than executing it, that you can then use as inputs to downstream tasks or operators. 9 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University
Control Flow DAGs are designed to be run many times, and multiple runs of them can happen in parallel. DAGs are parameterized, always including an interval they are "running for" (the data interval), but with other optional parameters as well. 10 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University
Triggering These dependencies are what make up the "edges" of the graph, and how Airflow works out which order to run your tasks in. By default, a task will wait for all of its upstream tasks to succeed before it runs, but this can be customized using features like Branching, LatestOnly, and Trigger Rules. 11 E6893 Big Data Analytics — Lecture 5 © CY Lin, Columbia University
Hiển thị 11 trang đầu. Tải tài liệu để xem đầy đủ.
Chưa có bình luận nào. Hãy là người đầu tiên!
Bình luận (0)
Chưa có bình luận nào. Hãy là người đầu tiên!