What is a macro in Sparkfllows?

General
1

A macro in Sparkflows is a variable that gets expanded into a string at runtime. Macros enable dynamic template generation, allowing you to insert execution-specific values into SQL queries, file paths, commands, and other task parameters.

They are especially useful when you want your workflows to adapt automatically based on execution dates or runtime context.

Macros allow you to:

  • Dynamically generate file paths (e.g., date-partitioned S3 paths)

  • Insert execution dates into SQL queries

  • Pass runtime variables to Spark or Bash commands

  • Build reusable and parameterized DAGs

For example, in a SQLOperator, you can inject execution dates into queries. Similarly, in a BashOperator, you can pass dynamic parameters into shell commands.

All predefined Airflow macros can be used in Sparkflows pipelines. These macros help dynamically generate paths, dates, and other runtime values inside your pipeline definitions.

Macros are written using double curly braces {{ }}.

s3://fre-sample-data/output/outputIngestion/dt={{ prev_ds_nodash }}

Here, {{ prev_ds_nodash }} dynamically inserts the previous execution date in YYYYMMDD format.