- DAG: a workflow which glues all tasks with inter-dependecies
- Operator: a template for a specific type of work to be executed. For example, BashOperator represents how to execute a bash script while PythonOperator represents how to execute a python function, etc.
- Sensor: a type of special operator which will only execute if a certain condition is met.
- Task: a parameterized instance of an operator/sensor which represents a unit of actual work to be executed.
- Plugin: an extension to allow users to easily extend Airflow with various custom hooks, operators, sensors, macros, and web views.
- Pools: concurrency limit configuration for a set of Airflow tasks
A celery queuecheck by scheduling a dummy task to every queue.
The “canary” DAG helps the oncall to answer the following questions:
- how long it takes for the Airflow scheduler to schedule the task (scheduled execution_time — current_time).
- how long it takes for celery worker to pick up the task.
- how long the task runs.
monitor 其任务的状态，以及整体队列的挤压情况
dag dependency部分缺失，以及整体的状态上有缺失，应该是已经有了。。。