org.apache.spark.SparkClassNotFoundException- [DATA_SOURCE_NOT_FOUND] Failed to find the data source- delta, When Running Read Incorta Node on Chidori Cluster

General
1

Problem

While submitting a PySpark job on Chidori, the job fails with the following error:

org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND]
Failed to find the data source: delta.

Cause

The Delta Lake libraries are not included in the Spark application, or the required Spark configurations for Delta Lake are missing.

Solution

Pass the Delta Lake JARs and the required Spark configurations during job submission.

--conf spark.jars=/opt/spark/incorta/delta-core_2.12-2.4.0.jar,/opt/spark/incorta/delta-storage-2.4.0.jar
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

Note: Ensure that the Delta Lake JAR version is compatible with the Spark and Scala versions being used.

Verification

After submitting the job with the above configurations, verify that:

  • The PySpark job starts successfully.
  • Spark is able to read from and write to Delta tables.
  • The DATA_SOURCE_NOT_FOUND error no longer occurs.