SensorsĪre derived from BaseSensorOperator and run a poke S3, a partition appearing in Hive, or a specific time of the day. Examples include a specific file landing in HDFS or Sensors are a certain type of operator that will keep running until aĬertain criterion is met. Transfer operators move data from one system to another Operators that performs an action, or tell another system to Operators allow for generation of certain types of tasks that become nodes in In details (such as output format and available flags) so if you want to rely on those in programmatic
For specific needs you can also use theĪirflow Command Line Interface (CLI) though it’s behaviour might change You can also use Airflow’s Public Interface via the Stable REST API (based on the On the other hand, classes and methods starting with _ (also knownĪs protected Python methods) and _ (also known as private Python methods) are not part of the PublicĪirflow Interface and might change at any time.
The classesĪnd functions mentioned below can be relied on to keep backwards-compatible signatures and behaviours within Provide a reusable set of functionality for external services or applications Airflow integrates with.Īll the ways above involve extending or using Airflow Python classes and functions. Provider packages - this is usually done by those who intend to Is usually done by users who manage Airflow instances.īundling custom Operators, Hooks, Plugins and releasing them together via Secrets, Timetables, Triggers, Listeners are all examples of such functionality. When writing new Plugins that extend Airflow’s functionality beyondĭAG building blocks. This can be done by DAG authors to add missing functionality in their DAGs or by those who write reusable custom operators for other DAG authors. When you are extending Airflow classes such as Operators and Hooks. Using Airflow Public Interfaces is needed when you want to interact with Airflow programmatically: What is not part of the Public Interface of Apache Airflow?.Using Public Interface to integrate with external services and applications.Using Public Interface to extend Airflow capabilities.Using the Public Interface for DAG Authors.Scalable: Airflow has a modular architecture and uses a message queue to talk to orchestrate an arbitrary number of workers.Parameterizing your scripts is built in the core of Airflow using powerful Jinja templating engine. Elegant: Airflow pipelines are lean and explicit.Extensible: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
Airflow logo code#
This allows for writting code that instantiate pipelines dynamically.
Airflow is a platform to programmaticaly author, schedule and monitor data pipelines.
0 kommentar(er)
