<!-- Source: https://docs.squirro.com/en/latest/getting/install/cluster-expansion.html -->
# Cluster Expansion

[Deprecated](../release/squirro-release-process.md#getting-squirro-process)

> **Note for administrators**
>
> End of support for expanding a Squirro cluster on Linux manually. This page remains available for reference only and is no longer maintained. Use Ansible for all installations and upgrades. See the [Install and Manage Squirro with Ansible](ansible/index.md#ansible) page.

> **Note**
>
> Squirro removed the Cluster Service (`sqclusterd`) in version 3.16.2, together with its `cluster.ini` configuration file and the `endpoint_discovery` and `db_endpoint_discovery` settings. The information on this page applies only to Squirro versions earlier than 3.16.2. For high-availability and multi-node deployments on Squirro 3.16.2 and later, visit the [Squirro Support website](https://go.squirro.com/support) and submit a technical support request.

This page details how to add cluster nodes to a Squirro installation. For base Linux installation, see the [Installing Squirro on Linux](linux.md#install-linux) page.

For background on Squirro cluster setups, see the [How Squirro Scales](../../technical/admin/operations/how-scales.md#admin-how-scales) page. That page provides an overview of Squirro components and their scaling considerations.

## Prerequisites

To add cluster nodes, first ensure that:

- The Linux machines are up and running, see the [System Requirements](system-requirements.md#install-system-requirements) page for the list of supported operating systems.
- The network is set up and all the machines that are to become part of the Squirro cluster can talk to each other.
- The firewalls are open between the machines with the documented ports accessible.
- The Squirro YUM repository is configured and accessible. In an enterprise environment, where this poses a problem, offline installation is available. In this case, visit the [Squirro Support website](https://go.squirro.com/support) and submit a technical support request.

## Overview

Any expansion of the cluster requires some work on the old and new nodes. This is outlined in the processes below by splitting the work up into sections, based on where the work is to be executed.

The process as described here involves some cluster downtime. It is possible to expand a Squirro cluster without any downtime involved - but that process requires a bit more planning and orchestration. If you need a downtime-free expansion, visit the [Squirro Support website](https://go.squirro.com/support) and submit a technical support request.

## Scaling Considerations

Cluster nodes and storage nodes can be scaled independently. You can add storage nodes without modifying the cluster node configuration, and vice versa.

**Adding storage nodes**

Add storage nodes one at a time. Adding multiple nodes simultaneously increases the risk of discovery and replication issues during cluster formation.

**Master-eligible node count**

To avoid split-brain situations, keep an odd number of master-eligible storage nodes (1, 3, 5, and so on). With an even number of nodes, the cluster may fail to elect a leader if the nodes become equally divided across a network partition. Squirro recommends following the Elasticsearch guidance on limiting the number of master-eligible nodes to a small, odd-numbered set. For more information, see the [Elasticsearch high-availability guide](https://www.elastic.co/guide/en/elasticsearch/reference/8.16/high-availability-cluster-small-clusters.html#high-availability-cluster-design-three-plus-nodes) on the Elasticsearch website.

## Storage Node Installation

The Squirro storage nodes are based on Elasticsearch. As such, some of the configurations needed for adding a storage node are found in the Elasticsearch configuration files.

**Process for New Storage Nodes**

Follow the steps below to add a new storage node.

1. Install/update the storage node package, as described in [Storage Node Installation](linux.md#install-linux-storage).
2. Apply the configuration from the previous storage nodes to the new one. Copy over the following setting from `/etc/elasticsearch/elasticsearch.yml`:

`cluster.name`

Make sure that this setting value is copied from the previous storage nodes to the new one - and not the other way around.

**Process for All Storage Nodes (Existing as Well as New)**

1. Allow the hosts to discover each other. Again in `/etc/elasticsearch/elasticsearch.yml` change the following settings:

   - Allow Elasticsearch to bind to a network interface by using the following config:

     `/etc/elasticsearch/elasticsearch.yml`

     ```yaml
     network.host: <server ip>,127.0.0.1
     ```

     This is a list of the server’s own IP addresses (as can be retrieved with `ip addr` for example), for example: `10.1.4.5`.
   - Set `discovery.seed_hosts` and `cluster.initial_master_nodes` to a list of all the storage nodes that have been set up. For example:

     `/etc/elasticsearch/elasticsearch.yml`

     ```yaml
     discovery.seed_hosts: ["<storagenode1 ip>", "<storagenode2 ip>", "<storagenode3 ip>"]
     cluster.initial_master_nodes: ["<storagenode1 ip>", "<storagenode2 ip>", "<storagenode3 ip>"]
     ```

     This is the easiest way to set up discovery and ensure all the Elasticsearch nodes can see each other. However, there are also other ways of configuring the discovery of the Elasticsearch nodes.

     This is documented by Elasticsearch in the [Discovery section of the Elasticsearch Manual](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html).
   - Also, optionally, you can set the node name to a friendlier name:

     `/etc/elasticsearch/elasticsearch.yml`

     ```yaml
     # User friendly node name
     node.name: test-node1
     ```
2. For new nodes remove the current Elasticsearch state:

   ```bash
   mv /var/lib/elasticsearch/nodes /tmp/
   ```

   > **Note**
   >
   > You can also remove this folder instead of moving it to `/tmp`. Moving it allows you to recover if you ran this on the wrong node.
3. Restart the service for the settings to take effect.

   ```bash
   systemctl restart elasticsearch
   ```
4. To verify the nodes discovered each other and formed a cluster, you can debug with this command:

   ```
   es_curl https://localhost:9200/_nodes?pretty    | less
   ```

   If successful, you should see the correct number of nodes in the output:

   ```json
   {
     "_nodes" : {
       "total" : 3,
       "successful" : 3,
       "failed" : 0
     }
   }
   ```
5. Set up the number of shards and number of replicas.
6. Modify `number_of_shards` and `number_of_replicas` in the templates. For more information, see the [Configuring Elasticsearch Templates](../../technical/admin/operations/managing-elasticsearch.md#admin-elasticsearch-templates) page. For multi-node storage setups, set `number_of_replicas` to `1` and `number_of_shards` to the number of Elasticsearch nodes.
7. Now update the shards and replicas settings of indices that were already present on the cluster before updating the templates using the curl command below. You can also selectively update the shards and replica settings of a particular index (instead of all indices) by replacing the `*` with the name of the index. This is a cluster wide setting and only needs to be done on one of the nodes of ES cluster.

   ```bash
   es_curl -XPUT https://127.0.0.1:9200/*/_settings -H "Content-Type: application/json" -d '{"index": {"number_of_replicas": 1}}'
   ```

## Cluster Node Installation

### Process for Each New Squirro Cluster Node Server

1. Install the cluster node package, as described in [Cluster Node Installation](linux.md#install-linux-nodes).
2. Ensure that each of the cluster node can talk to the Elasticsearch cluster (Squirro Storage Node). Change the config at `/etc/squirro/common.ini` to

   ```ini
   [index]
   es_index_servers = <storagenode1 ip>:9200,<storagenode2 ip>:9200,<storagenode3 ip>:9200
   ```
3. Allowlist all the Squirro Cluster nodes in the following nginx ipfilter files:

   - `/etc/nginx/conf.d/ipfilter-cluster-nodes.inc`
   - `/etc/nginx/conf.d/ipfilter-api-clients.inc`
   - `/etc/nginx/conf.d/ipfilter-monitoring.inc`.

     In each of these files include each IP address as follows:

     ```bash
     allow <clusternode1 ip>;
     allow <clusternode2 ip>;
     allow <clusternode2 ip>;
     ```

     Alternatively, the `allow` directive also accepts network addresses, for example `10.1.4.0/24` to allowlist an entire network.
4. Reload nginx at each of the cluster nodes.

   ```bash
   $ systemctl reload nginx
   ```
5. Set up the shared storage that the cluster nodes rely on, as described in the [Setting up Cluster Node Storage](#install-cluster-expansion-storage) section below.
6. Configure the relational database, as described in the [Relational Database Configuration](#install-cluster-expansion-database) section below.
7. Configure Redis, as described in the [Redis Configuration](#install-cluster-expansion-redis) section below.
8. Start all Squirro services on the new cluster node.

   ```bash
   $ squirro_start
   ```

For high-availability deployments where the relational database (MariaDB or PostgreSQL) or Redis is replicated across nodes, visit the [Squirro Support website](https://go.squirro.com/support) and submit a technical support request.

### Relational Database Configuration

The settings below apply to MariaDB, the default relational database. When using PostgreSQL instead, see the [Configure PostgreSQL as the Database Backend](postgresql.md#install-postgresql) page.

1. Give the node a unique identity within the cluster. The `/etc/mysql/conf.d/replication.cnf` file is installed with the following two settings commented out, so set both of them on every node:

   - `server_id`

     An integer that must be unique across the whole cluster. For example, use `10` for the first server in the cluster, `11` for the second, and so on.
   - `report_host`

     The name of the server as it is reported to the other hosts, for example `node01`.
2. Raise the limits on open files and maximum connections. Create the `/etc/mysql/conf.d/maxconnections.cnf` file with the following content:

   ```ini
   [mysqld]
   open_files_limit = 8192
   max_connections = 500
   ```

   Set `max_connections` higher depending on the number of cluster nodes. Squirro recommends at least 150 connections for each cluster node.
3. Restart MariaDB to apply both changes.

   ```bash
   $ systemctl restart mariadb
   ```

### Redis Configuration

All cluster nodes must reach the same Redis instances, one for storage and one for the cache. The `/etc/redis/redis.conf` and `/etc/redis/cache.conf` files are installed to listen on the local interface only, so extend the listening addresses on the server that runs Redis.

1. Edit both `/etc/redis/redis.conf` and `/etc/redis/cache.conf`, listing every cluster node:

   ```text
   bind 127.0.0.1 <clusternode1 ip> <clusternode2 ip> <clusternode3 ip>
   ```

   Where the cluster node addresses are not fixed, use `bind 0.0.0.0` instead, which configures Redis to listen on all network interfaces and IP addresses on the server.

   > **Warning**
   >
   > Redis must never be reachable from client or public networks. Restrict ports 6379 and 6380 to the cluster nodes at the network or security-group level, as described on the [System Requirements](system-requirements.md#install-system-requirements) page. Squirro configures password authentication for both Redis instances, but network isolation is the primary defense layer.
2. Restart both Redis services.

   ```bash
   $ systemctl restart redis-server
   $ systemctl restart redis-server-cache
   ```
3. Point every cluster node at that server by setting the `host` option of each `[redis*]` section in the `/etc/squirro/common.ini` file.

## Setting up Cluster Node Storage

In a multi-node cluster deployment, several directories must reside on a shared filesystem accessible from all cluster nodes, so the platform can read and write shared data regardless of which node handles a request.

For what is stored and how storage requirements vary by workload, see the [System Requirements](system-requirements.md#install-system-requirements) page. This section describes the directories to share and how to mount them.

### Shared Directories

The following directories must be present on the shared filesystem on all cluster nodes:

- **Binary document storage** (`/var/lib/squirro/storage/localfile`)

  Stores binary documents indexed by the platform, including PDF and Office files, along with associated assets such as images and thumbnails.
- **Pipelets** (`/var/lib/squirro/plumber`)

  Stores custom pipeline plugins deployed to the cluster.
- **Machine learning models** (`/var/lib/squirro/machinelearning`)

  Stores trained machine learning inference models. Each workflow creates a subdirectory under this path.
- **Guide files** (`/var/lib/squirro/topic/guide_files`)

  Stores guide files uploaded through the Squirro frontend.
- **Custom assets** (`/var/lib/squirro/topic/assets`)

  Stores custom CSS and other assets deployed to the cluster.
- **Custom widgets** (`/var/lib/squirro/topic/widgets`)

  Stores custom widget files deployed to the cluster.
- **Temporary files** (`/var/lib/squirro/temp`)

  Stores temporary files written during data ingestion and processing. Files are removed after processing completes.
- **Elasticsearch synonyms** (`/etc/elasticsearch/synonyms`)

  Stores custom synonym files loaded by Elasticsearch. This directory is created by the storage node package installer.

### Mounting the Shared Filesystem

The recommended approach is to keep the Squirro configuration files unchanged and use symbolic links to redirect the shared directories to the shared filesystem mount point.

For example, if the shared filesystem is mounted at `/mnt/squirro-shared`, create a symbolic link for each directory on every cluster node:

```bash
ln -s /mnt/squirro-shared/storage/localfile /var/lib/squirro/storage/localfile
ln -s /mnt/squirro-shared/plumber /var/lib/squirro/plumber
ln -s /mnt/squirro-shared/machinelearning /var/lib/squirro/machinelearning
ln -s /mnt/squirro-shared/topic/guide_files /var/lib/squirro/topic/guide_files
ln -s /mnt/squirro-shared/topic/assets /var/lib/squirro/topic/assets
ln -s /mnt/squirro-shared/topic/widgets /var/lib/squirro/topic/widgets
ln -s /mnt/squirro-shared/temp /var/lib/squirro/temp
```

Ensure the target directories exist on the shared filesystem before creating the symbolic links.

### SELinux Configuration for NFS

On systems with SELinux in enforcing mode, for example RHEL or CentOS, nginx requires an additional SELinux policy module to access files served from an NFS mount.

Run the following script on each cluster node to create and install the policy module:

```bash
mkdir -p /etc/squirro/selinux

cat > /etc/squirro/selinux/squirro-nginx-nfs.te << 'EOF'
module squirro-nginx-nfs 1.0;

require {
    type httpd_t;
    type nfs_t;
    class file { getattr open read };
}

allow httpd_t nfs_t:file { getattr open read };
EOF

checkmodule -M -m -o /etc/squirro/selinux/squirro-nginx-nfs.mod \
    /etc/squirro/selinux/squirro-nginx-nfs.te
semodule_package -o /etc/squirro/selinux/squirro-nginx-nfs.pp \
    -m /etc/squirro/selinux/squirro-nginx-nfs.mod
semodule -i /etc/squirro/selinux/squirro-nginx-nfs.pp
```

## Troubleshooting

### Network Drop Between Servers

This could be caused by a network monitoring tool closing all idle connections at periodic interval. In this cases, try lowering the TCP keep-alive used by the system and services:

Example, setting the value to 600s:

- Change `/proc/sys/net/ipv4/tcp_keepalive_time` value to 600:

```shell-session
# echo 600 > /proc/sys/net/ipv4/tcp_keepalive_time
```

- Change “tcp-keepalive” value to 600 in `/etc/redis/redis.conf`
- Add a new file /etc/sysctl.d/98-elasticsearch.conf with the following content:

  ```ini
  # lower keepalive settings to avoid elasticsearch cluster disconnects
  net.ipv4.tcp_keepalive_time = 600
  net.ipv4.tcp_keepalive_intvl = 60
  net.ipv4.tcp_keepalive_probes = 20
  ```