Handling java.lang.Interruption Exception in Canara Sparkflows Environment

Setup & Configuration
1

Incase if you are facing the Exception : java.lang.Interruption after executing the workflow then use the below spark configuration before executing the workflow inorder to resolve the issue :

Image_20251218_41339_029 PM (1)
Image_20251218_41339_029 PM (1)1920×1237 555 KB

–conf spark.sql.legacy.timeParserPolicy=LEGACY
–conf spark.sql.parquet.int96RebaseModeInRead=CORRECTED
–conf spark.sql.parquet.int96RebaseModeInWrite=CORRECTED
–executor-memory 20g
–executor-cores 4
–num-executors 6

What each config means :

  • spark.sql.legacy.timeParserPolicy=LEGACY
    → Allows backward-compatible date/time parsing (important for old timestamps)

  • spark.sql.parquet.int96RebaseModeInRead=CORRECTED
    → Correct handling of INT96 timestamps when reading Parquet

  • spark.sql.parquet.int96RebaseModeInWrite=CORRECTED
    → Correct handling of INT96 timestamps when writing Parquet

  • --executor-memory 20g
    → Each executor gets 20 GB RAM

  • --executor-cores 4
    → Each executor uses 4 CPU cores

  • --num-executors 6
    → Total 6 executors in the Spark job