AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
This one is a small quality of life improvement, and I don’t want to admit how many times I forgot the as dag:, or worse, had as dag: repeated. Auto-register DAGs used in a context manager (no more as dag: needed) map: To transform the parameters just before the task is run.įor more information on dynamic task mapping, see the new sections of the doc on Transforming Mapped Data, Combining upstream data (aka “zipping”), and Assigning multiple parameters to a non-TaskFlow operator. zip: To combine multiple things without cross-product.expand_kwargs: To assign multiple parameters to a non-TaskFlow operator.Dynamic task mapping now includes support for: There are a few subtlties as to what you need installed in the virtual env depending on which context variables you access, so be sure to read the how-to on using the ExternalPythonOperator More improvements to Dynamic Task Mapping (AIP-42) external_python ( python = '/opt/venvs/task_deps/bin/python' ) def my_task ( data_interval_start, data_interval_env ) print ( f 'Looking at data between ' ). To make this easier we have introduced (and the matching ExternalPythonOperator) that lets you run an python function as an Airflow task in a pre-configured virtual env, or even a whole different python version. Easier management of conflicting python dependencies using the new ExternalPythonOperatorĪs much as we wish all python libraries could be used happily together that sadly isn’t the world we live in, and sometimes there are conflicts when trying to install multiple python libraries in an Airflow install – right now we hear this a lot with dbt-core. That includes details on how datasets are identified (URIs), how you can depend on multiple datasets, and how to think about what a dataset is (hint: don’t include “date partitions” in a dataset, it’s higher level than that). We know that what exists right now won’t fit all use cases that people might wish for datasets, and in the coming minor releases (2.5, 2.6, etc.) we will expand and improve upon this foundation.ĭatasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write capability - in this release we are adding the foundational feature that we will build upon in the future - and it’s part of our goal to have smaller releases to get new features in your hands sooner!įor more information on datasets, see the documentation on Data-aware scheduling. With these two DAGs, the instant my_task finishes, Airflow will create the DAG run for the dataset-consumer workflow. First lets write a simple DAG with a task called my_task that produces a dataset called my-dataset:įrom airflow import Dataset dataset = Dataset ( uri = 'my-dataset' ) with DAG ( dag_id = 'dataset-consumer', schedule = ). If you are currently using ExternalTaskSensor or TriggerDagRunOperator you should take a look at datasets – in most cases you can replace them with something that will speed up the scheduling!īut enough talking, lets have a short example. What does this mean, exactly? This is a great new feature that lets DAG authors create smaller, more self-contained DAGs, which chain together into a larger data-based workflow. Airflow now has the ability to schedule DAGs based on other tasks updating datasets. □ Constraints: Data-aware scheduling (AIP-48) □ Docker Image: docker pull apache/airflow:2.4.0 That includes 46 new features, 39 improvements, 52 bug fixes, and several documentation changes. Apache Airflow 2.4.0 contains over 650 “user-facing” commits (excluding commits to providers or chart) and over 870 total.
0 Comments
Read More
Leave a Reply. |