# General **URL:** https://community.sparkflows.ai/c/general/4.md [Latest](https://community.sparkflows.ai/latest.md) · [Categories](https://community.sparkflows.ai/categories.md) · [Tags](https://community.sparkflows.ai/tags.md) --- ## [Welcome to Sparkflows! 👋](https://community.sparkflows.ai/t/welcome-to-sparkflows/5)
**Author:** [@system](https://community.sparkflows.ai/u/system)\ **Replies:** 0\ **Last updated:** [December 6, 2025, 9:06pm UTC](https://community.sparkflows.ai/t/welcome-to-sparkflows/5 "2025-12-06T21:06:33Z")
Welcome! Create an account to participate in discussions, personalize your profile, and connect with the community. Once you’ve joined, you can: Introduce yourself. Follow topics that interest you. Start discussions … --- ## [About the General category](https://community.sparkflows.ai/t/about-the-general-category/3)
**Author:** [@system](https://community.sparkflows.ai/u/system)\ **Replies:** 0
--- ## [Sparkflows Single Sign-On (SSO) login issue following the SSL certificate update](https://community.sparkflows.ai/t/sparkflows-single-sign-on-sso-login-issue-following-the-ssl-certificate-update/357)
**Author:** [@Shad](https://community.sparkflows.ai/u/Shad)\ **Replies:** 0\ **Last updated:** [September 17, 2026, 7:16am UTC](https://community.sparkflows.ai/t/sparkflows-single-sign-on-sso-login-issue-following-the-ssl-certificate-update/357 "2026-09-17T07:16:05Z")
When you update the SSL certificate of your Keycloak/SSO Provider with a certificate whose CA isn’t present in the Java keystore, then the authentication silently fails without providing any message. In order to solve t… --- ## [Handling multiple Livy URLs for HA](https://community.sparkflows.ai/t/handling-multiple-livy-urls-for-ha/355)
**Author:** [@Shad](https://community.sparkflows.ai/u/Shad)\ **Replies:** 1\ **Last updated:** [September 9, 2026, 6:10am UTC](https://community.sparkflows.ai/t/handling-multiple-livy-urls-for-ha/355 "2026-09-09T06:10:29Z")
We have deployed Livy services in a High Availability (HA) setup. For example, in the Prod environment, Livy is available on two nodes, with one node active and the other in standby mode. Currently, Sparkflows allows on… --- ## [SparkML workflow with xgboost are failing](https://community.sparkflows.ai/t/sparkml-workflow-with-xgboost-are-failing/354)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [August 7, 2026, 11:41am UTC](https://community.sparkflows.ai/t/sparkml-workflow-with-xgboost-are-failing/354 "2026-08-07T11:41:13Z")
Problem SparkML workflows using XGBoost may fail with the following error: java.lang.Exception: Error: Failed to execute the workflow: /tmp/libxgboost4j14342423119254343105.so: libgomp.so.1: cannot open shared object fi… --- ## [H2O Model Save to S3 Fails Due to SSL Certificate Hostname Mismatch](https://community.sparkflows.ai/t/h2o-model-save-to-s3-fails-due-to-ssl-certificate-hostname-mismatch/353)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [August 6, 2026, 12:43pm UTC](https://community.sparkflows.ai/t/h2o-model-save-to-s3-fails-due-to-ssl-certificate-hostname-mismatch/353 "2026-08-06T12:43:30Z")
Issue While saving an H2O model (MOJO) to an S3-compatible object store, the operation fails with the following exception: org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on s3a://... com.amazonaws.SdkClien… --- ## [Job failed, when saving the model in S3](https://community.sparkflows.ai/t/job-failed-when-saving-the-model-in-s3/352)
**Author:** [@Shad](https://community.sparkflows.ai/u/Shad)\ **Replies:** 0\ **Last updated:** [August 6, 2026, 12:39pm UTC](https://community.sparkflows.ai/t/job-failed-when-saving-the-model-in-s3/352 "2026-08-06T12:39:23Z")
Following is the exception I received. org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on s3a://mlflow-artifacts/prod/All\_Models\_Test/h2o\_S3\_test/6449cf60-a992-423a-bec1-aaba2643891a: com.amazonaws.SdkClien… --- ## [Sparkflows Installer \[macOS\] Fix: DMG Installer fails with "UnsupportedClassVersionError" (Homebrew Java 17)](https://community.sparkflows.ai/t/sparkflows-installer-macos-fix-dmg-installer-fails-with-unsupportedclassversionerror-homebrew-java-17/350)
**Author:** [@Daniel](https://community.sparkflows.ai/u/Daniel)\ **Replies:** 0\ **Last updated:** [August 3, 2026, 8:45am UTC](https://community.sparkflows.ai/t/sparkflows-installer-macos-fix-dmg-installer-fails-with-unsupportedclassversionerror-homebrew-java-17/350 "2026-08-03T08:45:40Z")
The Problem: While trying to install the application via the macOS DMG, the installer failed during the database creation step. A popup error appeared with the following message: Exception in thread “main” java.lang.Uns… --- ## [H2O Sparkling Water IllegalAccessError on Livy](https://community.sparkflows.ai/t/h2o-sparkling-water-illegalaccesserror-on-livy/349)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 30, 2026, 12:01pm UTC](https://community.sparkflows.ai/t/h2o-sparkling-water-illegalaccesserror-on-livy/349 "2026-07-30T12:01:49Z")
Issue During workflow execution on Livy, the workflow may fail with the following exception: Received an Exception: java.lang.IllegalAccessError: class ai.h2o.sparkling.backend.utils.RestCommunication (in unnamed modul… --- ## [Troubleshooting: Expired Client Certificate in k3s.yaml](https://community.sparkflows.ai/t/troubleshooting-expired-client-certificate-in-k3s-yaml/348)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 29, 2026, 10:56am UTC](https://community.sparkflows.ai/t/troubleshooting-expired-client-certificate-in-k3s-yaml/348 "2026-07-29T10:56:46Z")
Checking Kubernetes (k3s.yaml) Client Certificate Expiry The k3s.yaml file contains the client certificate used by kubectl to authenticate with the Kubernetes API server. You can check its validity period using the comma… --- ## [PySpark Workflow Submission Fails with SparkException: Exception thrown in awaitResult](https://community.sparkflows.ai/t/pyspark-workflow-submission-fails-with-sparkexception-exception-thrown-in-awaitresult/347)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 20, 2026, 11:53am UTC](https://community.sparkflows.ai/t/pyspark-workflow-submission-fails-with-sparkexception-exception-thrown-in-awaitresult/347 "2026-07-20T11:53:49Z")
Issue PySpark Workflow Submission Fails with SparkException: Exception thrown in awaitResult Error org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.SparkThreadUtils$.awaitR… --- ## [Spark Cluster Workflow Execution: No Browser Results](https://community.sparkflows.ai/t/spark-cluster-workflow-execution-no-browser-results/346)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 17, 2026, 12:27pm UTC](https://community.sparkflows.ai/t/spark-cluster-workflow-execution-no-browser-results/346 "2026-07-17T12:27:45Z")
Problem When executing workflows on a Spark cluster, the workflow completes or starts running, but no results are displayed in the Browser. Cause Sparkflows submits workflow jobs using spark-submit. The Spark driver is … --- ## [Pipeline Failure: Max retry reached Due to Dynamic Macro Usage](https://community.sparkflows.ai/t/pipeline-failure-max-retry-reached-due-to-dynamic-macro-usage/345)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 16, 2026, 11:20am UTC](https://community.sparkflows.ai/t/pipeline-failure-max-retry-reached-due-to-dynamic-macro-usage/345 "2026-07-16T11:20:49Z")
Issue Pipeline execution fails with a Max retry reached error. Cause When Sparkflows macros are used directly inside pipeline nodes, and the macro values change on every execution while the pipeline is scheduled to run … --- ## [ImportError: Unable to Import AzureOpenAI from OpenAI](https://community.sparkflows.ai/t/importerror-unable-to-import-azureopenai-from-openai/344)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 13, 2026, 10:53am UTC](https://community.sparkflows.ai/t/importerror-unable-to-import-azureopenai-from-openai/344 "2026-07-13T10:53:57Z")
Error ImportError: cannot import name 'AzureOpenAI' from 'openai' (/home/incorta/lib/python3.8/site-packages/openai/\_\_init\_\_.py) Reason The application is trying to import the AzureOpenAI client: from openai import Azu… --- ## [EMR Cluster Job Failure: Unable to Locate ‘delta’ Data Source](https://community.sparkflows.ai/t/emr-cluster-job-failure-unable-to-locate-delta-data-source/343)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 10, 2026, 8:14am UTC](https://community.sparkflows.ai/t/emr-cluster-job-failure-unable-to-locate-delta-data-source/343 "2026-07-10T08:14:50Z")
Problem While running a job on the EMR cluster, the following error is encountered: Failed to find the delta data source This indicates that the required Delta Lake JAR is either: Missing from the EMR cluster environ… --- ## [Inconsistent Execution of Airflow Pipeline](https://community.sparkflows.ai/t/inconsistent-execution-of-airflow-pipeline/342)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 6, 2026, 10:34am UTC](https://community.sparkflows.ai/t/inconsistent-execution-of-airflow-pipeline/342 "2026-07-06T10:34:24Z")
Problem: Pipeline is not running consistently in Airflow. Solution: There could be different reasons for this issue: The Background Event Trigger Thread is stalled. The Websocket doesn’t receive events in timely m… --- ## [Pipeline Trigger Delay Due to Concurrent Scheduling](https://community.sparkflows.ai/t/pipeline-trigger-delay-due-to-concurrent-scheduling/341)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 3, 2026, 11:26am UTC](https://community.sparkflows.ai/t/pipeline-trigger-delay-due-to-concurrent-scheduling/341 "2026-07-03T11:26:45Z")
Problem: Scheduler takes a few minutes to trigger all the pipelines if there are too many of them triggered at the same time. Solution: There could be different reasons for this issue like: The RAM allocated to Spark… --- ## [org.apache.spark.SparkClassNotFoundException- \[DATA\_SOURCE\_NOT\_FOUND\] Failed to find the data source- delta, When Running Read Incorta Node on Chidori Cluster](https://community.sparkflows.ai/t/org-apache-spark-sparkclassnotfoundexception-data-source-not-found-failed-to-find-the-data-source-delta-when-running-read-incorta-node-on-chidori-cluster/340)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 3, 2026, 4:27am UTC](https://community.sparkflows.ai/t/org-apache-spark-sparkclassnotfoundexception-data-source-not-found-failed-to-find-the-data-source-delta-when-running-read-incorta-node-on-chidori-cluster/340 "2026-07-03T04:27:36Z")
Problem While submitting a PySpark job on Chidori, the job fails with the following error: org.apache.spark.SparkClassNotFoundException: \[DATA\_SOURCE\_NOT\_FOUND\] Failed to find the data source: delta. Cause The Delta La… --- ## [Workflow Not Getting Pushed to GitHub](https://community.sparkflows.ai/t/workflow-not-getting-pushed-to-github/339)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 2, 2026, 9:11am UTC](https://community.sparkflows.ai/t/workflow-not-getting-pushed-to-github/339 "2026-07-02T09:11:19Z")
Problem The user pushes project changes from the Workflow Editor, but the latest changes are not reflected in the GitHub repository. Cause The user does not have sufficient write permissions to push changes to the repos… --- ## [ImportError: cannot import name 'ABCIndex' from 'pandas.core.dtypes.generic' While Submitting PySpark Job on Chidori](https://community.sparkflows.ai/t/importerror-cannot-import-name-abcindex-from-pandas-core-dtypes-generic-while-submitting-pyspark-job-on-chidori/338)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [July 2, 2026, 5:42am UTC](https://community.sparkflows.ai/t/importerror-cannot-import-name-abcindex-from-pandas-core-dtypes-generic-while-submitting-pyspark-job-on-chidori/338 "2026-07-02T05:42:59Z")
Reason Compatible version of Pandas is not available or the existing Pandas installation is corrupted/inconsistent, resulting in the following error: ImportError: cannot import name 'ABCIndex' from 'pandas.core.dtypes.g… --- ## [java.lang.IllegalAccessError: ai.h2o.sparkling.backend.utils.RestCommunication When Running H2O Job](https://community.sparkflows.ai/t/java-lang-illegalaccesserror-ai-h2o-sparkling-backend-utils-restcommunication-when-running-h2o-job/337)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [June 19, 2026, 11:11am UTC](https://community.sparkflows.ai/t/java-lang-illegalaccesserror-ai-h2o-sparkling-backend-utils-restcommunication-when-running-h2o-job/337 "2026-06-19T11:11:30Z")
Issue: When running a H2O job on Livy, the following exception is encountered: java.lang.IllegalAccessError: class ai.h2o.sparkling.backend.utils.RestCommunication (in unnamed module @0x480846cb) cannot access class sun… --- ## [What are the benefits of running Python workloads on EC2 through Sparkflows?](https://community.sparkflows.ai/t/what-are-the-benefits-of-running-python-workloads-on-ec2-through-sparkflows/336)
**Author:** [@Daniel](https://community.sparkflows.ai/u/Daniel)\ **Replies:** 0\ **Last updated:** [June 8, 2026, 7:08am UTC](https://community.sparkflows.ai/t/what-are-the-benefits-of-running-python-workloads-on-ec2-through-sparkflows/336 "2026-06-08T07:08:27Z")
This capability provides automated lifecycle management of EC2 resources, helping organizations optimize infrastructure usage and costs. By provisioning compute resources only when needed and automatically terminating th… --- ## [Can Sparkflows execute Python applications on Amazon EC2 instances as part of a pipeline?](https://community.sparkflows.ai/t/can-sparkflows-execute-python-applications-on-amazon-ec2-instances-as-part-of-a-pipeline/335)
**Author:** [@Daniel](https://community.sparkflows.ai/u/Daniel)\ **Replies:** 0\ **Last updated:** [June 8, 2026, 7:07am UTC](https://community.sparkflows.ai/t/can-sparkflows-execute-python-applications-on-amazon-ec2-instances-as-part-of-a-pipeline/335 "2026-06-08T07:07:37Z")
Yes. Sparkflows enables end-to-end execution of Python applications on Amazon EC2 instances within a pipeline. Users can automatically provision and start an EC2 instance, execute Python scripts stored in Amazon S3 using… --- ## [How can I control the number of concurrent DAG runs in Sparkflows?](https://community.sparkflows.ai/t/how-can-i-control-the-number-of-concurrent-dag-runs-in-sparkflows/334)
**Author:** [@Daniel](https://community.sparkflows.ai/u/Daniel)\ **Replies:** 0\ **Last updated:** [June 8, 2026, 7:03am UTC](https://community.sparkflows.ai/t/how-can-i-control-the-number-of-concurrent-dag-runs-in-sparkflows/334 "2026-06-08T07:03:06Z")
Sparkflows now supports the max\_active\_runs parameter through the AddDagArguments node, allowing users to define the maximum number of DAG runs that can execute concurrently. The default value is empty, which preserves t… --- ## [When refreshing schema from S3 seeing PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target](https://community.sparkflows.ai/t/when-refreshing-schema-from-s3-seeing-pkix-path-building-failed-sun-security-provider-certpath-suncertpathbuilderexception-unable-to-find-valid-certification-path-to-requested-target/329)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [June 4, 2026, 11:19am UTC](https://community.sparkflows.ai/t/when-refreshing-schema-from-s3-seeing-pkix-path-building-failed-sun-security-provider-certpath-suncertpathbuilderexception-unable-to-find-valid-certification-path-to-requested-target/329 "2026-06-04T11:19:39Z")
While refreshing the schema from S3, the application was unable to establish a trusted SSL connection because certificate validation failed. The JVM could not find a valid certificate chain for the S3 endpoint, resulting… --- ## [ValidationException: Unsupported Instance Type in EMR RunJobFlow](https://community.sparkflows.ai/t/validationexception-unsupported-instance-type-in-emr-runjobflow/328)
**Author:** [@neeraj](https://community.sparkflows.ai/u/neeraj)\ **Replies:** 0\ **Last updated:** [June 2, 2026, 1:45pm UTC](https://community.sparkflows.ai/t/validationexception-unsupported-instance-type-in-emr-runjobflow/328 "2026-06-02T13:45:18Z")
When submitting a pipeline on Airflow with an incorrect parameter for the Create EMR Cluster step, the logs will capture details as below: botocore.exceptions.ClientError: An error occurred (ValidationException) when ca… --- ## [Troubleshooting Job Failures After Upgrading Python from 3.8 to 3.9](https://community.sparkflows.ai/t/troubleshooting-job-failures-after-upgrading-python-from-3-8-to-3-9/327)
**Author:** [@Daniel](https://community.sparkflows.ai/u/Daniel)\ **Replies:** 0\ **Last updated:** [June 1, 2026, 6:09am UTC](https://community.sparkflows.ai/t/troubleshooting-job-failures-after-upgrading-python-from-3-8-to-3-9/327 "2026-06-01T06:09:52Z")
Troubleshooting Job Failures After Upgrading Python from 3.8 to 3.9 If you have recently upgraded your environment from Python 3.8 to Python 3.9 and are experiencing job failures, the issue may be related to outdated pac… --- ## [How to configure S3 in the Sparkflows](https://community.sparkflows.ai/t/how-to-configure-s3-in-the-sparkflows/325)
**Author:** [@Daniel](https://community.sparkflows.ai/u/Daniel)\ **Replies:** 0\ **Last updated:** [May 27, 2026, 5:37am UTC](https://community.sparkflows.ai/t/how-to-configure-s3-in-the-sparkflows/325 "2026-05-27T05:37:22Z")
Configuring S3 Storage Access for a Group This configuration can be performed only by a user with administrative privileges. Required Information Before proceeding, ensure the following S3 configuration details are avai… --- ## [How to Get Pipeline Executions by Pipeline ID](https://community.sparkflows.ai/t/how-to-get-pipeline-executions-by-pipeline-id/320)
**Author:** [@Daniel](https://community.sparkflows.ai/u/Daniel)\ **Replies:** 0\ **Last updated:** [April 27, 2026, 10:16am UTC](https://community.sparkflows.ai/t/how-to-get-pipeline-executions-by-pipeline-id/320 "2026-04-27T10:16:55Z")
Get Pipeline Executions by Pipeline ID: A Complete Guide Modern data platforms rely heavily on pipelines to orchestrate workflows, automate tasks, and ensure seamless data processing. Monitoring these pipelines is just a… --- ## [How to import a workflow via API using cURL command](https://community.sparkflows.ai/t/how-to-import-a-workflow-via-api-using-curl-command/318)
**Author:** [@Daniel](https://community.sparkflows.ai/u/Daniel)\ **Replies:** 0\ **Last updated:** [April 24, 2026, 9:08am UTC](https://community.sparkflows.ai/t/how-to-import-a-workflow-via-api-using-curl-command/318 "2026-04-24T09:08:42Z")
Importing Workflows via API Using cURL Managing workflows across projects often requires a quick and reliable way to import multiple workflow definitions. This guide walks you through using a simple API endpoint with cUR… [Next page](https://community.sparkflows.ai/c/general/4.md?page=1)