What are the common macros that can be used in the Sparkflows with examples

General
1

{{ ds }} returns the execution date in YYYY-MM-DD format. 2018-01-08

{{ ds_nodash }} returns the execution date in YYYYMMDD format. 20180108

{{ ts }} returns the execution timestamp in ISO format. 2018-01-01T00:00:00+00:00

Related variables:

  • {{ ts_nodash }} → 20180101T000000

  • {{ ts_nodash_with_tz }} → 20180101T000000+0000

{{ execution_date }} returns the execution date of the job

What are {{ prev_ds }} and {{ prev_ds_nodash }}?

  • {{ prev_ds }} → Previous execution date in YYYY-MM-DD

  • {{ prev_ds_nodash }} → Previous execution date in YYYYMMDD

If:

  • {{ ds }} = 2018-01-08

  • schedule_interval = @weeklyweekly

Then:

  • {{ prev_ds }} = 2018-01-01

What are {{ next_ds }} and {{ next_ds_nodash }}?

  • {{ next_ds }} → Next execution date in YYYY-MM-DD

  • {{ next_ds_nodash }} → Next execution date in YYYYMMDD

If:

  • {{ ds }} = 2018-01-01

  • schedule_interval = @weekly

Then:

  • {{ next_ds }} = 2018-01-08

What do yesterday and tomorrow macros return?

  • {{ yesterday_ds }} → Day before execution date (YYYY-MM-DD)

  • {{ yesterday_ds_nodash }} → Day before (YYYYMMDD)

  • {{ tomorrow_ds }} → Day after execution date (YYYY-MM-DD)

  • {{ tomorrow_ds_nodash }} → Day after (YYYYMMDD)

What are execution history macros?

  • {{ prev_execution_date }} → Previous execution date

  • {{ prev_execution_date_success }} → Previous successful DAG run date

  • {{ prev_start_date_success }} → Start date of previous successful DAG run

  • {{ next_execution_date }} → Next execution date

What objects are accessible in templates?

Airflow provides access to internal objects:

  • {{ dag }} → DAG object

  • {{ task }} → Task object

  • {{ task_instance }} or {{ ti }} → Task instance

  • {{ dag_run }} → DagRun object

  • {{ macros }} → Macros package reference

  • {{ conf }} → Airflow configuration object

  • {{ run_id }} → Current DAG run ID

  • {{ test_mode }} → Whether task is running in test mode