Pipeline Delay Due to Concurrent Scheduling

Setup & Configuration
1

Problem

When a large number of pipelines are scheduled to start at the same time, the scheduler may take several minutes to trigger all of them. This delay is typically caused by resource limitations or suboptimal Quartz scheduler configuration.

Possible Causes

  • Insufficient RAM allocated to the Sparkflows machine.
  • Low -Xmx value configured for the fire-ui server (default is 2.5 GB).
  • High CPU or memory utilization due to other processes running on the Sparkflows machine.
  • Quartz scheduler thread pool and trigger acquisition settings left at their default values.

Solution

1. Ensure Sufficient System Resources

For a large number of concurrent scheduled pipelines:

  • Allocate adequate RAM to the Sparkflows machine.
  • Ensure sufficient CPU cores are available.
  • Avoid running resource-intensive external processes on the same machine.

2. Increase JVM Heap Size

Increase the maximum heap size (-Xmx) for the fire-ui server.

Example: For triggering approximately 200 pipelines simultaneously, configure a higher -Xmx value than the default 2.5 GB. The recommended value depends on the overall machine specifications and workload.

3. Tune Quartz Scheduler Configuration

Update the following properties in:

<SPARKFLOWS_HOME>/conf/quartz.properties

org.quartz.threadPool.threadCount=20
org.quartz.scheduler.batchTriggerAcquisitionMaxCount=20
org.quartz.jobStore.acquireTriggersWithinLock=true

Configuration Details

Property Recommended Value Description
org.quartz.threadPool.threadCount 20 Number of scheduler threads available to execute jobs concurrently.
org.quartz.scheduler.batchTriggerAcquisitionMaxCount 20 Number of triggers acquired in a single scheduler cycle.
org.quartz.jobStore.acquireTriggersWithinLock true Acquires triggers within a database lock to improve consistency during concurrent scheduling.

Note: These values are recommended starting points. Tune them based on the machine’s CPU, memory, and workload characteristics.

Additional Information

For more information about Quartz scheduler tuning, refer to the official Quartz Scheduler documentation.