Handling multiple Livy URLs for HA

General
1

We have deployed Livy services in a High Availability (HA) setup. For example, in the Prod environment, Livy is available on two nodes, with one node active and the other in standby mode.

Currently, Sparkflows allows only a single Livy URL to be configured in the connection settings. As a result, when the active Livy node becomes unavailable and the standby node is promoted to active status, Sparkflows continues to reference the previous URL. This requires us to manually update the connection configuration each time a failover occurs.

We would appreciate your assistance in identifying a solution that enables automatic URL switching within Sparkflows during Livy failover events, eliminating the need for manual intervention, and ensuring seamless connectivity.

3

If you have multiple Livy servers/endpoints, the standard approach is to put a reverse proxy or load balancer in front of them and expose a single URL.

For example:

                ┌───────────────┐
                │ Sparkflows /  │
                │ Client        │
                └───────┬───────┘
                        │
               https://livy.company.com
                        │
              ┌─────────▼─────────┐
              │ Load Balancer /   │
              │ Reverse Proxy     │
              │                   │
              │ Health checks     │
              │ TLS termination   │
              │ Sticky sessions   │
              └───────┬─────┬──────┘
                      │     │
          ┌───────────┘     └───────────┐
          ▼                             ▼
   ┌──────────────┐              ┌──────────────┐
   │ Livy Server 1│              │ Livy Server 2│
   │ :8998        │              │ :8998        │
   └──────┬───────┘              └──────┬───────┘
          │                             │
          ▼                             ▼
       YARN /                         YARN /
       Kubernetes                     Kubernetes

The client only needs:

https://livy.company.com

instead of:

http://livy01:8998
http://livy02:8998
http://livy03:8998

Important Livy consideration

There is one major caveat: Livy is stateful from the client’s perspective.

A Livy session/batch is created on one Livy instance:

POST /batches

and subsequent requests refer to that Livy instance’s state:

GET /batches/{batchId}
GET /batches/{batchId}/state
GET /batches/{batchId}/log
DELETE /batches/{batchId}

Therefore, you should not blindly use round-robin load balancing unless your Livy deployment is designed to share/replicate state.

Recommended architecture

If your Livy instances are independent, use sticky sessions at the proxy/load-balancer layer:

                livy.company.com
                       │
                ┌──────▼──────┐
                │ Load Balancer│
                └──────┬──────┘
                       │
            ┌──────────┴──────────┐
            │                     │
         Livy-1                Livy-2
         :8998                  :8998

The LB maintains affinity:

Client A ───────────────► Livy-1
                          │
POST /batches             │
GET /batches/123 ────────►│
GET /batches/123/state ──►│

while another client can go to Livy-2:

Client B ───────────────► Livy-2

HAProxy

HAProxy is often a better fit if you need explicit session persistence and health checks.

Conceptually:

frontend livy
    bind *:443 ssl
    default_backend livy_servers

backend livy_servers
    balance roundrobin
    cookie LIVY_SERVER insert indirect nocache

    server livy01 livy01:8998 check cookie livy01
    server livy02 livy02:8998 check cookie livy02
    server livy03 livy03:8998 check cookie livy03

The browser/client receives a cookie identifying the Livy backend, so subsequent requests remain associated with the same Livy server.