PySpark Workflow Submission Fails with SparkException: Exception thrown in awaitResult

General
1

Issue

PySpark Workflow Submission Fails with SparkException: Exception thrown in awaitResult

Error

org.apache.spark.SparkException: Exception thrown in awaitResult:
    at org.apache.spark.util.SparkThreadUtils$.awaitResult(SparkThreadUtils.scala:56)
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:310)
    at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:506)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:265)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:938)

Cause

The required Spark configuration properties were missing from the Livy connection, causing the PySpark workflow submission to fail.

Resolution

Add the following Spark properties to the Livy Connection configuration:

spark.driver.extraJavaOptions=-Dcom.amazonaws.sdk.disableCertChecking=true
spark.executor.extraJavaOptions=-Dcom.amazonaws.sdk.disableCertChecking=true
spark.pyspark.python=/usr/local/bin/python3.x
spark.pyspark.driver.python=/usr/local/bin/python3.x

After adding these properties, save the Livy connection and resubmit the PySpark workflow.

Note: Replace /usr/local/bin/python3.x with the actual Python executable path available on your cluster (for example, /usr/bin/python3 or /usr/local/bin/python3.9). Ensure that the same Python path is available on all cluster nodes.